Oct 31 03:16:50.968919 ip-10-0-138-238 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 03:16:50.969130 ip-10-0-138-238 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 03:16:50.969220 ip-10-0-138-238 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 03:16:50.969755 ip-10-0-138-238 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 03:17:01.012379 ip-10-0-138-238 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 03:17:01.012391 ip-10-0-138-238 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 97aefef74dfa4de7a6d72ee88f7db9d7 -- Oct 31 03:18:44.322533 ip-10-0-138-238 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:18:44.739882 ip-10-0-138-238 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:18:44.740502 ip-10-0-138-238 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:18:44.740502 ip-10-0-138-238 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:18:44.740502 ip-10-0-138-238 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:18:44.744163 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744053 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.744163 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744163 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744167 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744171 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744174 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744177 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744179 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744182 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744185 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744188 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744192 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744196 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744200 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744202 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744205 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744209 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744212 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744215 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744217 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744220 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.744244 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744222 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744244 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744249 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744254 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744259 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744262 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744265 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744269 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744271 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744274 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744277 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744279 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744282 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744284 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744287 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744289 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744292 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744294 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744296 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.744677 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744299 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744301 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744303 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744306 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744308 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744311 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744314 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744318 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744322 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744326 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744329 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744331 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744334 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744337 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744339 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744342 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744345 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744347 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744349 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.745104 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744352 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744354 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744357 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744359 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744362 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744364 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744366 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744369 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744371 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744373 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744375 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744378 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744380 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744383 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744385 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744387 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744390 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744392 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744395 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744398 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.745565 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744400 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744403 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744405 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744408 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744411 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744413 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744417 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744419 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744422 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744424 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744427 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744429 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744432 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744434 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744436 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744439 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744441 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744443 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744446 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744448 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.746053 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744451 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744453 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744456 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744458 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744462 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744464 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744466 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.744469 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745283 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745291 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745294 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745297 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745300 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745304 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745307 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745309 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745312 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745315 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745318 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.746539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745321 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745323 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745326 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745329 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745331 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745334 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745339 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745343 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745346 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745349 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745351 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745354 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745357 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745359 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745362 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745365 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745367 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745370 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745372 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.746983 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745375 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745377 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745380 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745382 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745385 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745387 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745390 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745393 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745396 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745399 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745401 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745404 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745406 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745409 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745411 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745414 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745416 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745419 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745421 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.747440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745424 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745426 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745429 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745432 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745434 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745437 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745439 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745442 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745444 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745447 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745449 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745452 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745454 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745457 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745460 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745462 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745466 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745469 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745472 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.747875 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745474 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745477 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745480 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745482 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745484 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745487 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745490 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745493 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745495 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745498 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745500 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745503 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745505 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745507 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745510 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745512 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745515 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745517 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745520 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745522 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.748368 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745525 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745527 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745530 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745532 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745534 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745537 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745542 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745544 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745547 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745549 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745552 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745554 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745556 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745559 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745562 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745565 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745568 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.745571 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746114 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746119 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.748874 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746122 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746125 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746128 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746130 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746133 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746135 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746138 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746140 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746143 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746145 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746148 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746150 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746153 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746155 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746158 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746160 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746163 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746165 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746168 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746171 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.749376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746173 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746176 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746178 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746181 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746183 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746186 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746188 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746191 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746194 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746196 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746199 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746202 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746204 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746206 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746209 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746211 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746214 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746216 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746219 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746221 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.749902 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746242 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746244 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746247 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746249 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746252 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746254 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746257 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746259 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746261 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746264 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746267 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746270 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746272 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746275 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746277 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746280 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746282 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746285 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746287 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.750539 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746293 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746297 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746300 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746303 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746306 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746308 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746311 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746313 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746316 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746318 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746320 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746323 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746325 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746335 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746338 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746340 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746343 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746346 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746348 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.751032 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746351 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746354 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746356 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746359 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746361 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746364 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746367 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746370 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746372 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746375 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746377 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746380 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746382 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746385 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746388 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746391 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746394 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746396 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746399 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746401 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.751556 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746404 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746406 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746408 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746411 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746416 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.746419 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747414 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747425 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747432 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747437 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747441 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747444 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747449 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747453 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747457 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747460 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747463 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747466 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747470 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747473 2575 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747475 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747478 2575 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747481 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:18:44.752073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747484 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747489 2575 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747493 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747496 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747500 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747503 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747507 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747510 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747513 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747516 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747519 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747522 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747525 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747528 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747531 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747535 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747538 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747541 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747544 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747547 2575 flags.go:64] FLAG: --enable-server="true" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747552 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747556 2575 flags.go:64] FLAG: --event-burst="100" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747559 2575 flags.go:64] FLAG: --event-qps="50" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747562 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:18:44.752698 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747566 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747569 2575 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747573 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747576 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747579 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747582 2575 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747585 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747588 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747592 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747595 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747599 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747602 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747605 2575 flags.go:64] FLAG: --feature-gates="" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747610 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747613 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747616 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747619 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747622 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747625 2575 flags.go:64] FLAG: --help="false" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747628 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747632 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747635 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747638 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:18:44.753459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747642 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747645 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747648 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747651 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747654 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747657 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747660 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747664 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747667 2575 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747669 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747672 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747675 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747678 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747681 2575 flags.go:64] FLAG: --lock-file="" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747683 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747686 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747689 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747695 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747697 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747702 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747705 2575 flags.go:64] FLAG: --logging-format="text" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747707 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747711 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747714 2575 flags.go:64] FLAG: --manifest-url="" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747717 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:18:44.754047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747722 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747725 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747729 2575 flags.go:64] FLAG: --max-pods="110" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747732 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747735 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747738 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747741 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747744 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747747 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747750 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747757 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747760 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747764 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747767 2575 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747770 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747773 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747776 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747779 2575 flags.go:64] FLAG: --port="10250" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747782 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747785 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01d83ca4bf958a366" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747788 2575 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747791 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747794 2575 flags.go:64] FLAG: --register-node="true" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747797 2575 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747801 2575 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747803 2575 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:18:44.754795 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747807 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747810 2575 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747814 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747817 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747820 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747824 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747827 2575 flags.go:64] FLAG: --runonce="false" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747830 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747833 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747836 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747838 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747841 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747845 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747848 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747851 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747854 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747856 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747859 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747864 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747867 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747870 2575 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747873 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747878 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747882 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747885 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:18:44.755895 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747889 2575 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747892 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747895 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747898 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747900 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747903 2575 flags.go:64] FLAG: --v="2" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747908 2575 flags.go:64] FLAG: --version="false" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747914 2575 flags.go:64] FLAG: --vmodule="" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747918 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.747921 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748057 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748061 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748064 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748067 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748070 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748072 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748075 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748077 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748080 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748082 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748084 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748087 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.756982 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748089 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748092 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748094 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748098 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748101 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748103 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748106 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748108 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748111 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748113 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748116 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748118 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748121 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748123 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748127 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748130 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748133 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748137 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748140 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.758037 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748142 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748145 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748147 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748149 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748152 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748155 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748157 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748160 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748163 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748165 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748168 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748170 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748172 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748175 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748177 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748179 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748183 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748185 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748187 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748190 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.758895 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748192 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748195 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748197 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748200 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748202 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748205 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748207 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748210 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748212 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748215 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748219 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748221 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748238 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748243 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748247 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748251 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748255 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748258 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748261 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.759607 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748264 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748266 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748269 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748271 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748274 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748276 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748279 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748282 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748284 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748288 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748291 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748293 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748296 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748298 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748301 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748303 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748306 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748308 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748311 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.760254 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748313 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748317 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748320 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748323 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748327 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748329 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748332 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748334 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748337 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748339 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748342 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748344 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748347 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748350 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748352 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748355 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.761026 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748357 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.748385 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748421 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748425 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748427 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748430 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748432 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748436 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748439 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748442 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748444 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748447 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748450 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748452 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748455 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.761568 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748457 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748460 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748462 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748465 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748467 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748472 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748476 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748479 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748481 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748484 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748487 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748489 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748492 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748495 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748497 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748501 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748503 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748506 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748508 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.761916 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748511 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748513 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748515 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748518 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748520 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748524 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748526 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748529 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748531 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748534 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748536 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748539 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748541 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748544 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748546 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748549 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748551 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748553 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748558 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748560 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.762468 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748563 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748565 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748568 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748570 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748573 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748575 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748578 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748580 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748586 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748590 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748593 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748596 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748598 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748601 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748604 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748606 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748609 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748612 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748615 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.762944 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748617 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748620 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748622 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748625 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748627 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748630 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748632 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748635 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748637 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748640 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748642 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748646 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748648 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748650 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748653 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748655 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748658 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748660 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748663 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748665 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.763554 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748668 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748671 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748673 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748677 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748680 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748682 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748685 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748687 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748689 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748692 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748696 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748699 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748701 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748704 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.748706 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.757162 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.757186 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.757209 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.757217 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:18:44.764020 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757332 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757340 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757345 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757349 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757356 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757362 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757367 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757372 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757376 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757380 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757384 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757389 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757393 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757397 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757402 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757406 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757410 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757414 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757418 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.764542 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757422 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757426 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757431 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757437 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757443 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757448 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757452 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757457 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757461 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757465 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757471 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757476 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757481 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757486 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757490 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757494 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757499 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757504 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757508 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.764988 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757512 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757517 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757521 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757526 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757530 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757534 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757539 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757543 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757547 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757551 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757555 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757560 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757564 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757568 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757573 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757577 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757581 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757585 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757589 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757593 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.765519 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757598 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757603 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757607 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757612 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757618 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757622 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757626 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757630 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757634 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757639 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757643 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757647 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757651 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757655 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757660 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757664 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757668 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757672 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757676 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.766045 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757681 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757685 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757690 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757694 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757698 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757702 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757706 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757710 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757715 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757719 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757723 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757727 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757731 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757736 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757740 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757744 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757749 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757754 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757758 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757763 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.766555 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757767 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757772 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757776 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757780 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757784 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757789 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757793 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757797 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.757801 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.757843 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758077 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758086 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758091 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758095 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758099 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.767139 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758106 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758113 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758119 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758124 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758128 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758132 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758137 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758141 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758145 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758149 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758154 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758158 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758164 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758168 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758172 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758177 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758187 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758192 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758197 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.767739 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758201 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758205 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758209 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758213 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758217 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758221 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758244 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758248 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758252 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758257 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758260 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758265 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758269 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758273 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758277 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758281 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758285 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758289 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758293 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758297 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.768485 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758301 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758305 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758310 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758314 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758318 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758322 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758334 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758339 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758343 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758347 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758353 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758358 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758362 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758366 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758370 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758374 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758379 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758382 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758386 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.768948 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758390 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758395 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758399 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758403 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758407 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758411 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758415 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758420 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758423 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758428 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758431 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758435 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758442 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758447 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758451 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758456 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758460 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758464 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758468 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758472 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.769440 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758476 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758480 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758485 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758489 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758494 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758498 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758502 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758507 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758511 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758515 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758518 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758523 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758527 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758531 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758535 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758539 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758543 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758547 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758551 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758555 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.769937 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758559 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758563 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758567 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.758608 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758662 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758669 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758673 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758677 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758681 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758685 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758689 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758694 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758699 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758703 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758707 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758711 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:44.770446 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758716 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758720 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758724 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758729 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758733 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758737 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758741 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758745 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758749 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758753 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758757 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758761 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758765 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758769 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758773 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758777 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758782 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758786 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758790 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758794 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:44.770841 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758798 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758803 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758807 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758810 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758814 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758818 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758823 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758828 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758831 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758835 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758839 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758844 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758848 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758853 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758857 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758862 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758866 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758870 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758874 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:44.771329 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758878 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758882 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758886 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758890 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758894 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758898 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758901 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758905 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758908 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758912 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758916 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758920 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758924 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758928 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758932 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758937 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758941 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758948 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758953 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758957 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:44.771755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758962 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758966 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758970 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758974 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758978 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758982 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758987 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758991 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.758996 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759000 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759004 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759008 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759012 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759016 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759020 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759024 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759028 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759033 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759040 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:44.772214 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759044 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759048 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759052 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759057 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759061 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759065 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759069 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759073 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759077 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759082 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759086 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759090 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759095 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759098 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759107 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:44.759113 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.759199 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.763064 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 03:18:44.772823 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.764179 2575 server.go:1014] "Starting client certificate rotation" Oct 31 03:18:44.773495 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.764256 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:18:44.773495 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.764291 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:18:44.773495 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.767596 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:18:44.776506 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.776476 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:18:44.787706 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.787682 2575 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:18:44.791276 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.791259 2575 log.go:25] "Validated CRI v1 image API" Oct 31 03:18:44.792156 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.792131 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:18:44.795520 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.795496 2575 fs.go:135] Filesystem UUIDs: map[5b85a941-1bf5-48de-8b4c-0034d8475172:/dev/nvme0n1p3 6aea7901-7c47-426c-b91a-7ea692c015c5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 31 03:18:44.795581 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.795519 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 03:18:44.800767 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.800651 2575 manager.go:217] Machine: {Timestamp:2026-10-31 03:18:44.799352218 +0000 UTC m=+0.358855179 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098115 MemoryCapacity:32811995136 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6c5c2aa45d595ad30523d4f5a443 SystemUUID:ec2a6c5c-2aa4-5d59-5ad3-0523d4f5a443 BootID:97aefef7-4dfa-4de7-a6d7-2ee88f7db9d7 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16405999616 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405995520 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:69:2b:0a:db:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:69:2b:0a:db:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:53:0b:ed:f7:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32811995136 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:18:44.800767 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.800766 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:18:44.800900 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.800885 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:18:44.804798 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.804768 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:18:44.804937 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.804800 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-238.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:18:44.804986 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.804940 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:18:44.804986 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.804948 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:18:44.804986 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.804977 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:18:44.805066 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.804996 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:18:44.805786 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.805776 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:18:44.809361 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.809349 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:18:44.809488 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.809479 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:18:44.810175 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.810165 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:18:44.810217 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.810180 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:18:44.810217 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.810193 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:18:44.810217 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.810203 2575 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:18:44.810217 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.810213 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:18:44.810592 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.810576 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:18:44.812560 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.812538 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:18:44.817968 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.817955 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:18:44.819649 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.819623 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:18:44.819731 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.819678 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:18:44.819936 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.819922 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:18:44.819983 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.819944 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:18:44.819983 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.819954 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:18:44.819983 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.819962 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:18:44.820116 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820001 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:18:44.820116 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820013 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:18:44.820580 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820559 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:18:44.820734 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820726 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:18:44.820765 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820737 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:18:44.820765 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820744 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:18:44.820765 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820754 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:18:44.820765 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.820765 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:18:44.821627 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.821617 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:18:44.821627 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.821627 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:18:44.825497 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.825473 2575 server.go:1263] "Started kubelet" Oct 31 03:18:44.825598 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.825553 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:18:44.825667 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.825581 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:18:44.825726 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.825695 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:18:44.826243 ip-10-0-138-238 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:18:44.827040 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.826932 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:18:44.828383 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.828368 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:18:44.830522 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.830505 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 03:18:44.833270 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.833256 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:18:44.834450 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.834434 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:18:44.834676 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.834656 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:44.835218 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835199 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:18:44.835218 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835199 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:18:44.835218 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835240 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:18:44.835409 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835253 2575 factory.go:55] Registering systemd factory Oct 31 03:18:44.835409 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835268 2575 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:18:44.835409 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835327 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:18:44.835409 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835336 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:18:44.835537 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835498 2575 factory.go:153] Registering CRI-O factory Oct 31 03:18:44.835537 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835509 2575 factory.go:223] Registration of the crio container factory successfully Oct 31 03:18:44.835587 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835548 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:18:44.835587 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835562 2575 factory.go:103] Registering Raw factory Oct 31 03:18:44.835587 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.835576 2575 manager.go:1199] Started watching for new ooms in manager Oct 31 03:18:44.836467 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.836452 2575 manager.go:319] Starting recovery of all containers Oct 31 03:18:44.847822 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.847796 2575 manager.go:324] Recovery completed Oct 31 03:18:44.854128 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.854113 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:44.855728 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.855707 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:44.855803 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.855740 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:44.855803 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.855750 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:44.856354 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.856342 2575 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:18:44.856354 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.856353 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:18:44.856432 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.856368 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:18:44.856432 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.856405 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:44.857090 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.857077 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:44.857123 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.857103 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:44.857123 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.857113 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:44.858542 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.858530 2575 policy_none.go:50] "Start" Oct 31 03:18:44.858592 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.858547 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:18:44.858592 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.858561 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:18:44.860489 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.860478 2575 policy_none.go:44] "Start" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.901532 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.910452 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.911762 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.911781 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.911801 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.911836 2575 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.912839 2575 manager.go:341] "Starting Device Plugin manager" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.912871 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.912883 2575 server.go:87] "Starting device plugin registration server" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.913260 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.913270 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.913440 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.913512 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.913521 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.914051 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:18:44.919446 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.914104 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:44.972564 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:44.972539 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-238.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 03:18:44.972564 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.972545 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-238.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 03:18:44.973604 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:44.972532 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-238.us-west-1.compute.internal.18e37f0261770069 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-238.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-238.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:44.825440361 +0000 UTC m=+0.384943322,LastTimestamp:2026-10-31 03:18:44.825440361 +0000 UTC m=+0.384943322,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-238.us-west-1.compute.internal,}" Oct 31 03:18:45.012042 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.011972 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal"] Oct 31 03:18:45.012042 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.012045 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:45.012982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.012967 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:45.013043 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.012998 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:45.013043 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.013009 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:45.014033 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.014020 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:45.014742 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.014726 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:45.014827 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.014756 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:45.014827 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.014770 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:45.014827 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.014802 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.015363 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.015346 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:45.015492 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.015477 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.015550 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.015515 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:45.016008 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.015992 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:45.016087 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.016009 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:45.016087 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.016018 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:45.016087 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.016029 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:45.016087 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.016035 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:45.016087 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.016049 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:45.018549 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.018535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.018624 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.018560 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:45.019255 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.019241 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:45.019334 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.019266 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:45.019334 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.019278 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:45.051818 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.051799 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.053115 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.053099 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:18:45.053184 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.053169 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dzv5b" Oct 31 03:18:45.054442 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.054426 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.075174 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.075092 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-238.us-west-1.compute.internal.18e37f0263452149 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-238.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-238.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-238.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:44.855726409 +0000 UTC m=+0.415229372,LastTimestamp:2026-10-31 03:18:44.855726409 +0000 UTC m=+0.415229372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-238.us-west-1.compute.internal,}" Oct 31 03:18:45.075174 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.075166 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 03:18:45.075352 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.075176 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 31 03:18:45.075352 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.075345 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-238.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 03:18:45.075447 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.075407 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 03:18:45.134594 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.134571 2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-238.us-west-1.compute.internal\" not found" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.136785 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.136768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de19d3008b27f6bbf6579b9645a2ecb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"0de19d3008b27f6bbf6579b9645a2ecb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.136862 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.136800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.136862 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.136827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.237284 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.237256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.237413 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.237285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.237413 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.237312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de19d3008b27f6bbf6579b9645a2ecb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"0de19d3008b27f6bbf6579b9645a2ecb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.237413 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.237345 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.237413 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.237355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.237413 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.237378 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de19d3008b27f6bbf6579b9645a2ecb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"0de19d3008b27f6bbf6579b9645a2ecb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.280343 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.279874 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-238.us-west-1.compute.internal\" not found" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.334834 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.334796 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:45.335885 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.335869 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:45.335960 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.335899 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:45.335960 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.335909 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:45.335960 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.335942 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.356164 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.356147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.359312 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.359296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.393285 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.393262 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:45.393285 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.393287 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-238.us-west-1.compute.internal\": node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:45.507608 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.507582 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:45.608659 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.608580 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:45.709303 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.709272 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:45.764215 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:45.764191 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 03:18:45.764638 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.764400 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3bc042c2ecd546b6b1e906ab067c5a8-f96d67daba3dd493.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-238.us-west-1.compute.internal.18e37f0263452149\": read tcp 10.0.138.238:33346->3.18.80.101:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-238.us-west-1.compute.internal.18e37f0263452149 default 4687 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-238.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-238.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-238.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:44 +0000 UTC,LastTimestamp:2026-10-31 03:18:45.014739977 +0000 UTC m=+0.574242942,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-238.us-west-1.compute.internal,}" Oct 31 03:18:45.809509 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.809481 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:45.910316 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:45.910245 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:46.011349 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.011308 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:46.058950 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.058900 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:13:45 +0000 UTC" deadline="2028-06-01 07:12:10.01061888 +0000 UTC" Oct 31 03:18:46.058950 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.058944 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13899h53m23.95167788s" Oct 31 03:18:46.111483 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.111451 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:46.119020 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.118993 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:46.119909 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.119892 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:46.120011 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.119929 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:46.120011 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.119944 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:46.212028 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.211954 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:46.312531 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.312501 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:46.340874 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.340846 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:46.413201 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.413153 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:46.423212 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.423198 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:46.470376 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:46.470284 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87725f65687ad3eca42b0f7a2fe25aa4.slice/crio-16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8 WatchSource:0}: Error finding container 16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8: Status 404 returned error can't find the container with id 16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8 Oct 31 03:18:46.470545 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:46.470525 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de19d3008b27f6bbf6579b9645a2ecb.slice/crio-01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4 WatchSource:0}: Error finding container 01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4: Status 404 returned error can't find the container with id 01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4 Oct 31 03:18:46.476980 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.476963 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:18:46.513959 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.513921 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:18:46.518373 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.518355 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:46.600540 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.600519 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:46.635674 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.635648 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:46.699875 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.699847 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:18:46.700902 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.700887 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:18:46.761974 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.761946 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:18:46.812224 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.812185 2575 apiserver.go:52] "Watching apiserver" Oct 31 03:18:46.834715 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.834680 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 03:18:46.916640 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.916581 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"87725f65687ad3eca42b0f7a2fe25aa4","Type":"ContainerStarted","Data":"16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8"} Oct 31 03:18:46.917793 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.917772 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"0de19d3008b27f6bbf6579b9645a2ecb","Type":"ContainerStarted","Data":"01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4"} Oct 31 03:18:46.924746 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.924727 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:18:46.925047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.925031 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-qgw8p","kube-system/konnectivity-agent-vgq6v","openshift-cluster-node-tuning-operator/tuned-hhs74","openshift-image-registry/node-ca-ntzrg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-2r4ff","kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal","openshift-multus/multus-jw6nt","openshift-ovn-kubernetes/ovnkube-node-gwqdn","kube-system/global-pull-secret-syncer-5ngpx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb","openshift-multus/network-metrics-daemon-z2hw8","openshift-dns/node-resolver-kh444","openshift-network-diagnostics/network-check-target-wdw6h"] Oct 31 03:18:46.929373 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.929353 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.931753 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.931711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:46.931849 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.931813 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:18:46.933993 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.933966 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:46.936388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.936369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:46.938675 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.938648 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:46.938759 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.938736 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:18:46.941220 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.941199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:46.943545 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.943525 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jw6nt" Oct 31 03:18:46.946536 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.945780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-bin\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.946536 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.945829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-script-lib\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.946536 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.945899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-log-socket\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.947724 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.947700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-systemd-units\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.947809 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.947757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-netns\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.947809 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.947792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-var-lib-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.947929 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.947862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.947929 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.947900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.948038 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.947967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzp8s\" (UniqueName: \"kubernetes.io/projected/cf4f6953-b84a-482d-b685-bd2addcbafde-kube-api-access-vzp8s\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.948038 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:46.948139 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c807d9-708e-4527-8e56-11749a26ca61-tmp-dir\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:46.948139 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7scj\" (UniqueName: \"kubernetes.io/projected/bdd9600b-d5c9-4add-ae37-48aca3854510-kube-api-access-z7scj\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:46.948238 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-slash\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.948238 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:46.948359 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-env-overrides\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.948359 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948268 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-netd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.948501 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6c807d9-708e-4527-8e56-11749a26ca61-hosts-file\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:46.948501 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhzp\" (UniqueName: \"kubernetes.io/projected/c6c807d9-708e-4527-8e56-11749a26ca61-kube-api-access-phhzp\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:46.948611 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-node-log\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.948611 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-kubelet\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.948718 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-ovn\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.949047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.948796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-etc-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.949389 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.949360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-config\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.949492 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.949407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf4f6953-b84a-482d-b685-bd2addcbafde-ovn-node-metrics-cert\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.949563 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.949496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-systemd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.949563 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.949536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdd9600b-d5c9-4add-ae37-48aca3854510-iptables-alerter-script\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:46.949671 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.949565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdd9600b-d5c9-4add-ae37-48aca3854510-host-slash\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:46.949671 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.949593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:46.950142 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.950122 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:46.952707 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.952686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:46.952817 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:46.952762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:18:46.955096 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.955078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:46.957383 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.957358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:46.987515 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.987381 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:18:46.987515 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.987424 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:18:46.987515 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.987453 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:18:46.987515 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.987433 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:18:46.987515 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.987477 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:18:46.987779 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.987539 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sv7k2\"" Oct 31 03:18:46.987779 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.987625 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:18:46.991134 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.991030 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:18:46.991134 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.991103 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:18:46.991283 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.991198 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xz7gs\"" Oct 31 03:18:46.993296 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.993278 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:18:46.993531 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.993516 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:18:46.993615 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.993602 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:18:46.993666 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.993606 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67ksm\"" Oct 31 03:18:46.998857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.998838 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:18:46.998857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.998853 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:18:46.998990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.998927 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:18:46.998990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.998971 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4ln2\"" Oct 31 03:18:46.998990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.998978 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:18:46.998990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:46.998979 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:18:47.000317 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.000298 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:18:47.000610 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.000597 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9x6rv\"" Oct 31 03:18:47.005361 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.005343 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:18:47.005428 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.005354 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:18:47.005428 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.005417 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:18:47.005640 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.005628 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltpjd\"" Oct 31 03:18:47.006803 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.006791 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:18:47.007099 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.007085 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:18:47.007154 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.007097 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:18:47.007265 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.007251 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4kwqf\"" Oct 31 03:18:47.036092 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.036055 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:18:47.036513 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.036495 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:18:47.036666 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.036650 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:18:47.036865 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.036847 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xmnbh\"" Oct 31 03:18:47.036933 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.036871 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4wd7m\"" Oct 31 03:18:47.036933 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.036879 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:18:47.036933 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.036906 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:18:47.038656 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.038641 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:18:47.038713 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.038693 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnllh" Oct 31 03:18:47.050850 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.050825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-sys\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.050949 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.050857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-systemd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.050949 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.050879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdd9600b-d5c9-4add-ae37-48aca3854510-host-slash\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:47.050949 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.050897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppr9\" (UniqueName: \"kubernetes.io/projected/93cfa49d-0426-4d23-ad84-50f12272a871-kube-api-access-sppr9\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.051073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.050959 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.051073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.051073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051038 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdd9600b-d5c9-4add-ae37-48aca3854510-host-slash\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:47.051073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051034 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-systemd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.051237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-kubernetes\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.051237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-socket-dir-parent\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.051237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-script-lib\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.051237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-netns\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051543 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-netns\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-kubelet\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-etc-kubernetes\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7scj\" (UniqueName: \"kubernetes.io/projected/bdd9600b-d5c9-4add-ae37-48aca3854510-kube-api-access-z7scj\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-multus-daemon-config\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.051921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-modprobe-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-tmp\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-system-cni-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-hostroot\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-lib-modules\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpcf\" (UniqueName: \"kubernetes.io/projected/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-kube-api-access-xzpcf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6c807d9-708e-4527-8e56-11749a26ca61-hosts-file\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad09c972-be06-48b1-84b0-a25fc1c645ba-agent-certs\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:47.054604 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-dbus\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-cnibin\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-systemd\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-kubelet\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-ovn\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-multus\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-device-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-script-lib\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-host\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-tuned\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-etc-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-serviceca\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052866 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6c807d9-708e-4527-8e56-11749a26ca61-hosts-file\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052927 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052944 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-kubelet\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.055388 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.052958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdh2\" (UniqueName: \"kubernetes.io/projected/1687ced2-8f63-4900-9b49-275a8104e159-kube-api-access-rsdh2\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053054 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-ovn\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-registration-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdd9600b-d5c9-4add-ae37-48aca3854510-iptables-alerter-script\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-bin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-conf-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-var-lib-kubelet\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysconfig\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-bin\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-log-socket\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-systemd-units\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-var-lib-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trx4z\" (UniqueName: \"kubernetes.io/projected/9ec0a926-6081-4a94-916a-2e27400b06a9-kube-api-access-trx4z\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-cni-binary-copy\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-socket-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.056071 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.053934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-sys-fs\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054002 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdd9600b-d5c9-4add-ae37-48aca3854510-iptables-alerter-script\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054155 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-bin\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzp8s\" (UniqueName: \"kubernetes.io/projected/cf4f6953-b84a-482d-b685-bd2addcbafde-kube-api-access-vzp8s\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054200 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-log-socket\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054240 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-systemd-units\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054284 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-var-lib-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c807d9-708e-4527-8e56-11749a26ca61-tmp-dir\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-system-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-netns\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-run\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-slash\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054082 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-etc-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-env-overrides\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.056648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054917 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c807d9-708e-4527-8e56-11749a26ca61-tmp-dir\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.054995 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055029 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055089 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-slash\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-host\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-multus-certs\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwvt\" (UniqueName: \"kubernetes.io/projected/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kube-api-access-qlwvt\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-netd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-env-overrides\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055551 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-netd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-cnibin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhzp\" (UniqueName: \"kubernetes.io/projected/c6c807d9-708e-4527-8e56-11749a26ca61-kube-api-access-phhzp\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-node-log\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad09c972-be06-48b1-84b0-a25fc1c645ba-konnectivity-ca\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:47.057127 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055884 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-node-log\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-config\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf4f6953-b84a-482d-b685-bd2addcbafde-ovn-node-metrics-cert\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mqds\" (UniqueName: \"kubernetes.io/projected/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-kube-api-access-7mqds\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.055998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-kubelet-config\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.056034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-k8s-cni-cncf-io\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.056126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-conf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.056308 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:18:47.057660 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.056396 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-config\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.059833 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.059815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf4f6953-b84a-482d-b685-bd2addcbafde-ovn-node-metrics-cert\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.117938 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.117906 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:47.117938 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.117933 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:47.117938 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.117944 2575 projected.go:196] Error preparing data for projected volume kube-api-access-25l6m for pod openshift-network-diagnostics/network-check-target-wdw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:47.118140 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.118005 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m podName:f15c9c88-bfe3-4acd-8047-9c3d9cd0296d nodeName:}" failed. No retries permitted until 2026-10-31 03:18:47.61798739 +0000 UTC m=+3.177490357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-25l6m" (UniqueName: "kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m") pod "network-check-target-wdw6h" (UID: "f15c9c88-bfe3-4acd-8047-9c3d9cd0296d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:47.120429 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.120405 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7scj\" (UniqueName: \"kubernetes.io/projected/bdd9600b-d5c9-4add-ae37-48aca3854510-kube-api-access-z7scj\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:47.120900 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.120874 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzp8s\" (UniqueName: \"kubernetes.io/projected/cf4f6953-b84a-482d-b685-bd2addcbafde-kube-api-access-vzp8s\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.121370 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.121344 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhzp\" (UniqueName: \"kubernetes.io/projected/c6c807d9-708e-4527-8e56-11749a26ca61-kube-api-access-phhzp\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:47.157167 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad09c972-be06-48b1-84b0-a25fc1c645ba-konnectivity-ca\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:47.157167 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mqds\" (UniqueName: \"kubernetes.io/projected/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-kube-api-access-7mqds\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-kubelet-config\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-k8s-cni-cncf-io\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-conf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-sys\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppr9\" (UniqueName: \"kubernetes.io/projected/93cfa49d-0426-4d23-ad84-50f12272a871-kube-api-access-sppr9\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157305 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-kubelet-config\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157357 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-sys\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157389 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-conf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-k8s-cni-cncf-io\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.157433 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-kubernetes\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157448 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-kubernetes\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157452 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-socket-dir-parent\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-socket-dir-parent\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-kubelet\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157534 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-etc-kubernetes\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-multus-daemon-config\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157625 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-kubelet\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-modprobe-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157711 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-etc-kubernetes\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-tmp\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-system-cni-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-hostroot\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-lib-modules\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpcf\" (UniqueName: \"kubernetes.io/projected/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-kube-api-access-xzpcf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad09c972-be06-48b1-84b0-a25fc1c645ba-agent-certs\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157950 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad09c972-be06-48b1-84b0-a25fc1c645ba-konnectivity-ca\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-dbus\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-cnibin\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-systemd\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158050 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-system-cni-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-multus\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.157768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-modprobe-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-device-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158090 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-lib-modules\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-host\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-tuned\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-serviceca\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:47.158857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158182 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158213 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-dbus\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158201 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-multus-daemon-config\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158244 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-cnibin\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158204 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-host\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158341 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-device-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-hostroot\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-systemd\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdh2\" (UniqueName: \"kubernetes.io/projected/1687ced2-8f63-4900-9b49-275a8104e159-kube-api-access-rsdh2\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158450 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-registration-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.158501 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-bin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-conf-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158564 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-multus\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-var-lib-kubelet\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.159645 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysconfig\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-registration-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.158699 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:47.658679686 +0000 UTC m=+3.218182640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trx4z\" (UniqueName: \"kubernetes.io/projected/9ec0a926-6081-4a94-916a-2e27400b06a9-kube-api-access-trx4z\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-cni-binary-copy\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-socket-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158774 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158928 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-serviceca\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-sys-fs\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.158995 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-conf-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.159048 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159029 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-sys-fs\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-system-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.159103 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret podName:3eefb160-a377-431f-be0a-c2bea6ecb352 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:47.659086552 +0000 UTC m=+3.218589527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret") pod "global-pull-secret-syncer-5ngpx" (UID: "3eefb160-a377-431f-be0a-c2bea6ecb352") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159098 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-system-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysconfig\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.160237 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159229 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-var-lib-kubelet\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-netns\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-socket-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-run\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159284 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159360 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-bin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159374 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-run\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159412 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-netns\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-host\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-multus-certs\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159530 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-host\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwvt\" (UniqueName: \"kubernetes.io/projected/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kube-api-access-qlwvt\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-cnibin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159598 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-cni-binary-copy\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159596 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-multus-certs\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159666 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-cnibin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.160774 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.161279 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.159782 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.161279 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.160519 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-tmp\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.161279 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.160622 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-tuned\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.161279 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.160863 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad09c972-be06-48b1-84b0-a25fc1c645ba-agent-certs\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:47.225225 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.225187 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mqds\" (UniqueName: \"kubernetes.io/projected/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-kube-api-access-7mqds\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.225567 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.225546 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppr9\" (UniqueName: \"kubernetes.io/projected/93cfa49d-0426-4d23-ad84-50f12272a871-kube-api-access-sppr9\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.225666 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.225590 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdh2\" (UniqueName: \"kubernetes.io/projected/1687ced2-8f63-4900-9b49-275a8104e159-kube-api-access-rsdh2\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.226004 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.225978 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trx4z\" (UniqueName: \"kubernetes.io/projected/9ec0a926-6081-4a94-916a-2e27400b06a9-kube-api-access-trx4z\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:47.226111 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.226092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpcf\" (UniqueName: \"kubernetes.io/projected/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-kube-api-access-xzpcf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.226156 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.226128 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwvt\" (UniqueName: \"kubernetes.io/projected/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kube-api-access-qlwvt\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.252459 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.252385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kh444" Oct 31 03:18:47.259500 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.259480 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:18:47.263076 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.263055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:18:47.263461 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.263437 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c807d9_708e_4527_8e56_11749a26ca61.slice/crio-c22d8d4598865ed32963a35212cc2a10673b9d9fbb9984199c6adf4cd6cdaaf3 WatchSource:0}: Error finding container c22d8d4598865ed32963a35212cc2a10673b9d9fbb9984199c6adf4cd6cdaaf3: Status 404 returned error can't find the container with id c22d8d4598865ed32963a35212cc2a10673b9d9fbb9984199c6adf4cd6cdaaf3 Oct 31 03:18:47.268466 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.268441 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4f6953_b84a_482d_b685_bd2addcbafde.slice/crio-437e8d737b8b666cb4aceb772a4327c2d99f84bbe2721c1799c57d7e1ce90a81 WatchSource:0}: Error finding container 437e8d737b8b666cb4aceb772a4327c2d99f84bbe2721c1799c57d7e1ce90a81: Status 404 returned error can't find the container with id 437e8d737b8b666cb4aceb772a4327c2d99f84bbe2721c1799c57d7e1ce90a81 Oct 31 03:18:47.272864 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.272841 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd9600b_d5c9_4add_ae37_48aca3854510.slice/crio-e293a71db3bd5f844f1fe493ed4af34894e79aaf3abd77b346b8fc75dedbd983 WatchSource:0}: Error finding container e293a71db3bd5f844f1fe493ed4af34894e79aaf3abd77b346b8fc75dedbd983: Status 404 returned error can't find the container with id e293a71db3bd5f844f1fe493ed4af34894e79aaf3abd77b346b8fc75dedbd983 Oct 31 03:18:47.274265 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.274227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jw6nt" Oct 31 03:18:47.277206 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.277190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:18:47.283773 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.283747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:18:47.285492 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.285468 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cfa49d_0426_4d23_ad84_50f12272a871.slice/crio-dba80bdeb3e79704bc274388bf14fccf7d5b4981c72c4d0f2c95b5505c96e3a3 WatchSource:0}: Error finding container dba80bdeb3e79704bc274388bf14fccf7d5b4981c72c4d0f2c95b5505c96e3a3: Status 404 returned error can't find the container with id dba80bdeb3e79704bc274388bf14fccf7d5b4981c72c4d0f2c95b5505c96e3a3 Oct 31 03:18:47.287668 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.287571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:18:47.287730 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.287717 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6c13c0_234d_4042_a31c_ac2c374fd1b3.slice/crio-53e025546c07d72f5d9533e39a5f13c0fba1524ee57f8f3f0d8de768ae7b67f2 WatchSource:0}: Error finding container 53e025546c07d72f5d9533e39a5f13c0fba1524ee57f8f3f0d8de768ae7b67f2: Status 404 returned error can't find the container with id 53e025546c07d72f5d9533e39a5f13c0fba1524ee57f8f3f0d8de768ae7b67f2 Oct 31 03:18:47.290637 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.290476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:18:47.293131 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.293112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:47.294755 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.294731 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1687ced2_8f63_4900_9b49_275a8104e159.slice/crio-77907b880c28799f8c97c0150d5ca25dbbbe3c1534e2e5cbb8eaed6ed712c17c WatchSource:0}: Error finding container 77907b880c28799f8c97c0150d5ca25dbbbe3c1534e2e5cbb8eaed6ed712c17c: Status 404 returned error can't find the container with id 77907b880c28799f8c97c0150d5ca25dbbbe3c1534e2e5cbb8eaed6ed712c17c Oct 31 03:18:47.298365 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.298340 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfceaec_d546_4c9e_ac02_e66eeeea477a.slice/crio-fcd73e7ab4ad36809a42fd963afd3fbd9d2a4ccd86d02336ca45de5495430799 WatchSource:0}: Error finding container fcd73e7ab4ad36809a42fd963afd3fbd9d2a4ccd86d02336ca45de5495430799: Status 404 returned error can't find the container with id fcd73e7ab4ad36809a42fd963afd3fbd9d2a4ccd86d02336ca45de5495430799 Oct 31 03:18:47.300651 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.300627 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39f2c1f_e94d_4a91_aabd_08b7094aaa38.slice/crio-1f33e7e0bac9bbeef9df66997be4efb666594f9a16811172a7be3c2997b430c4 WatchSource:0}: Error finding container 1f33e7e0bac9bbeef9df66997be4efb666594f9a16811172a7be3c2997b430c4: Status 404 returned error can't find the container with id 1f33e7e0bac9bbeef9df66997be4efb666594f9a16811172a7be3c2997b430c4 Oct 31 03:18:47.311261 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:18:47.311231 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad09c972_be06_48b1_84b0_a25fc1c645ba.slice/crio-4c3ac7e10b8a12eea0997eefb37a76c9fdd4453eb1a87cc0077a9d620241c91e WatchSource:0}: Error finding container 4c3ac7e10b8a12eea0997eefb37a76c9fdd4453eb1a87cc0077a9d620241c91e: Status 404 returned error can't find the container with id 4c3ac7e10b8a12eea0997eefb37a76c9fdd4453eb1a87cc0077a9d620241c91e Oct 31 03:18:47.664962 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.664880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:47.665226 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.665077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:47.665355 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.665333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:47.665595 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.665571 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:47.665677 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.665599 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:47.665677 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.665613 2575 projected.go:196] Error preparing data for projected volume kube-api-access-25l6m for pod openshift-network-diagnostics/network-check-target-wdw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:47.665836 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.665686 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m podName:f15c9c88-bfe3-4acd-8047-9c3d9cd0296d nodeName:}" failed. No retries permitted until 2026-10-31 03:18:48.665664005 +0000 UTC m=+4.225166974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-25l6m" (UniqueName: "kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m") pod "network-check-target-wdw6h" (UID: "f15c9c88-bfe3-4acd-8047-9c3d9cd0296d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:47.665836 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.665817 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:47.665952 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.665873 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:48.665857076 +0000 UTC m=+4.225360043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:47.666013 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.665958 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:47.666013 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:47.666004 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret podName:3eefb160-a377-431f-be0a-c2bea6ecb352 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:48.665991025 +0000 UTC m=+4.225493983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret") pod "global-pull-secret-syncer-5ngpx" (UID: "3eefb160-a377-431f-be0a-c2bea6ecb352") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:47.920793 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.920696 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgq6v" event={"ID":"ad09c972-be06-48b1-84b0-a25fc1c645ba","Type":"ContainerStarted","Data":"4c3ac7e10b8a12eea0997eefb37a76c9fdd4453eb1a87cc0077a9d620241c91e"} Oct 31 03:18:47.922721 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.922687 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hhs74" event={"ID":"b39f2c1f-e94d-4a91-aabd-08b7094aaa38","Type":"ContainerStarted","Data":"1f33e7e0bac9bbeef9df66997be4efb666594f9a16811172a7be3c2997b430c4"} Oct 31 03:18:47.924706 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.924675 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ntzrg" event={"ID":"ba6c13c0-234d-4042-a31c-ac2c374fd1b3","Type":"ContainerStarted","Data":"53e025546c07d72f5d9533e39a5f13c0fba1524ee57f8f3f0d8de768ae7b67f2"} Oct 31 03:18:47.926358 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.926330 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qgw8p" event={"ID":"bdd9600b-d5c9-4add-ae37-48aca3854510","Type":"ContainerStarted","Data":"e293a71db3bd5f844f1fe493ed4af34894e79aaf3abd77b346b8fc75dedbd983"} Oct 31 03:18:47.927775 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.927745 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kh444" event={"ID":"c6c807d9-708e-4527-8e56-11749a26ca61","Type":"ContainerStarted","Data":"c22d8d4598865ed32963a35212cc2a10673b9d9fbb9984199c6adf4cd6cdaaf3"} Oct 31 03:18:47.929255 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.929211 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" event={"ID":"3cfceaec-d546-4c9e-ac02-e66eeeea477a","Type":"ContainerStarted","Data":"fcd73e7ab4ad36809a42fd963afd3fbd9d2a4ccd86d02336ca45de5495430799"} Oct 31 03:18:47.930669 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.930636 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerStarted","Data":"77907b880c28799f8c97c0150d5ca25dbbbe3c1534e2e5cbb8eaed6ed712c17c"} Oct 31 03:18:47.932181 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.932134 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jw6nt" event={"ID":"93cfa49d-0426-4d23-ad84-50f12272a871","Type":"ContainerStarted","Data":"dba80bdeb3e79704bc274388bf14fccf7d5b4981c72c4d0f2c95b5505c96e3a3"} Oct 31 03:18:47.935103 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:47.935078 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"437e8d737b8b666cb4aceb772a4327c2d99f84bbe2721c1799c57d7e1ce90a81"} Oct 31 03:18:48.041644 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.041578 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:47 +0000 UTC" deadline="2028-06-16 17:19:35.74008982 +0000 UTC" Oct 31 03:18:48.041644 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.041640 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14270h0m47.69845414s" Oct 31 03:18:48.677105 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.677058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:48.677296 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.677117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:48.677296 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.677160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:48.677430 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677399 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:48.677430 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677421 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:48.677570 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677434 2575 projected.go:196] Error preparing data for projected volume kube-api-access-25l6m for pod openshift-network-diagnostics/network-check-target-wdw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:48.677570 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677497 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m podName:f15c9c88-bfe3-4acd-8047-9c3d9cd0296d nodeName:}" failed. No retries permitted until 2026-10-31 03:18:50.677477604 +0000 UTC m=+6.236980573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-25l6m" (UniqueName: "kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m") pod "network-check-target-wdw6h" (UID: "f15c9c88-bfe3-4acd-8047-9c3d9cd0296d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:48.677697 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677583 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:48.677697 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677629 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:50.677615957 +0000 UTC m=+6.237118922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:48.677697 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677686 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:48.677853 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.677723 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret podName:3eefb160-a377-431f-be0a-c2bea6ecb352 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:50.677712265 +0000 UTC m=+6.237215215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret") pod "global-pull-secret-syncer-5ngpx" (UID: "3eefb160-a377-431f-be0a-c2bea6ecb352") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:48.914796 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.914762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:48.914977 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.914904 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:18:48.915046 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.914997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:48.915117 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.915093 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:18:48.915179 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:48.915163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:48.915263 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:48.915245 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:18:49.043213 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:49.043159 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:47 +0000 UTC" deadline="2028-05-08 22:13:27.699198884 +0000 UTC" Oct 31 03:18:49.043213 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:49.043212 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13338h54m38.655992614s" Oct 31 03:18:49.965201 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:49.964865 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"0de19d3008b27f6bbf6579b9645a2ecb","Type":"ContainerStarted","Data":"da286635cdd55b852f86e81ee02173ce9a832cab14e01d8bb6299821a76c1251"} Oct 31 03:18:50.696318 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.696202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:50.696318 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.696277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:50.696832 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.696324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:50.696832 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696583 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:50.696832 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696607 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:50.696832 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696621 2575 projected.go:196] Error preparing data for projected volume kube-api-access-25l6m for pod openshift-network-diagnostics/network-check-target-wdw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:50.696832 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696687 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m podName:f15c9c88-bfe3-4acd-8047-9c3d9cd0296d nodeName:}" failed. No retries permitted until 2026-10-31 03:18:54.696667066 +0000 UTC m=+10.256170019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-25l6m" (UniqueName: "kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m") pod "network-check-target-wdw6h" (UID: "f15c9c88-bfe3-4acd-8047-9c3d9cd0296d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:50.696832 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696772 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:50.696832 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696812 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:54.696797933 +0000 UTC m=+10.256300886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:50.697090 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696855 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:50.697090 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.696877 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret podName:3eefb160-a377-431f-be0a-c2bea6ecb352 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:54.696870092 +0000 UTC m=+10.256373042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret") pod "global-pull-secret-syncer-5ngpx" (UID: "3eefb160-a377-431f-be0a-c2bea6ecb352") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:50.915187 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.915156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:50.915411 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.915384 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:50.915516 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.915469 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:18:50.915516 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.915500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:50.915624 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.915600 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:18:50.915684 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:50.915642 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:18:50.984851 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.984809 2575 generic.go:356] "Generic (PLEG): container finished" podID="87725f65687ad3eca42b0f7a2fe25aa4" containerID="9804aabb435e85f48b75047d13dd156bd7d3a2ff969d9ef2a88a94ae664b3a5e" exitCode=0 Oct 31 03:18:50.985426 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:50.985395 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"87725f65687ad3eca42b0f7a2fe25aa4","Type":"ContainerDied","Data":"9804aabb435e85f48b75047d13dd156bd7d3a2ff969d9ef2a88a94ae664b3a5e"} Oct 31 03:18:51.113887 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:51.113827 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" podStartSLOduration=4.113803486 podStartE2EDuration="4.113803486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:50.09254793 +0000 UTC m=+5.652050902" watchObservedRunningTime="2026-10-31 03:18:51.113803486 +0000 UTC m=+6.673306463" Oct 31 03:18:51.993529 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:51.993488 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"87725f65687ad3eca42b0f7a2fe25aa4","Type":"ContainerStarted","Data":"9831e94b1a1d3f944b5fd44b3ac49067cb98f084fc399debdff33a635f334462"} Oct 31 03:18:51.997479 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:51.996765 2575 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="00b84820fc469e6bdbec4fbb0fa727f7abe8717062325efae61a8de72565d861" exitCode=0 Oct 31 03:18:51.997479 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:51.996825 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerDied","Data":"00b84820fc469e6bdbec4fbb0fa727f7abe8717062325efae61a8de72565d861"} Oct 31 03:18:52.255515 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:52.255460 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" podStartSLOduration=5.255438087 podStartE2EDuration="5.255438087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:52.121123226 +0000 UTC m=+7.680626191" watchObservedRunningTime="2026-10-31 03:18:52.255438087 +0000 UTC m=+7.814941062" Oct 31 03:18:52.912565 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:52.912529 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:52.912745 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:52.912581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:52.912745 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:52.912719 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:18:52.912854 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:52.912772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:52.912906 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:52.912856 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:18:52.912973 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:52.912953 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:18:53.007585 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:53.007547 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgq6v" event={"ID":"ad09c972-be06-48b1-84b0-a25fc1c645ba","Type":"ContainerStarted","Data":"de3e536e66a439394fae44802208423d13b3f977acbd243ab3f2353055da6999"} Oct 31 03:18:54.014472 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.014396 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ntzrg" event={"ID":"ba6c13c0-234d-4042-a31c-ac2c374fd1b3","Type":"ContainerStarted","Data":"f1c2ba1a4109116760d6ed22cdf3bc3945af6f7935a39aa026dfc999d068fa76"} Oct 31 03:18:54.116505 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.116468 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:54.117144 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.117125 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:18:54.130802 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.130740 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vgq6v" podStartSLOduration=3.201105531 podStartE2EDuration="8.130720417s" totalImagesPullingTime="4.929614886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:53.125165083 +0000 UTC m=+8.684668054" watchObservedRunningTime="2026-10-31 03:18:54.130720417 +0000 UTC m=+9.690223388" Oct 31 03:18:54.247743 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.247695 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ntzrg" podStartSLOduration=2.048417868 podStartE2EDuration="8.247674921s" totalImagesPullingTime="6.199257053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:54.131029165 +0000 UTC m=+9.690532137" watchObservedRunningTime="2026-10-31 03:18:54.247674921 +0000 UTC m=+9.807177890" Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.732480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.732558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.732590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.732795 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.732820 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.732834 2575 projected.go:196] Error preparing data for projected volume kube-api-access-25l6m for pod openshift-network-diagnostics/network-check-target-wdw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.732919 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m podName:f15c9c88-bfe3-4acd-8047-9c3d9cd0296d nodeName:}" failed. No retries permitted until 2026-10-31 03:19:02.732900842 +0000 UTC m=+18.292403793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-25l6m" (UniqueName: "kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m") pod "network-check-target-wdw6h" (UID: "f15c9c88-bfe3-4acd-8047-9c3d9cd0296d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.733055 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.733094 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:02.733079968 +0000 UTC m=+18.292582918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.733163 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:54.733298 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.733195 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret podName:3eefb160-a377-431f-be0a-c2bea6ecb352 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:02.733184681 +0000 UTC m=+18.292687633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret") pod "global-pull-secret-syncer-5ngpx" (UID: "3eefb160-a377-431f-be0a-c2bea6ecb352") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:54.913519 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.913472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:54.913692 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.913619 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:18:54.913754 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.913697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:54.913816 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.913790 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:18:54.913897 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:54.913873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:54.913988 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:54.913970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:18:55.020324 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:55.020246 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" event={"ID":"3cfceaec-d546-4c9e-ac02-e66eeeea477a","Type":"ContainerStarted","Data":"8d9634070097dec8a454b2d4173802d5bd44c3637970c8bccd981acf5eb43120"} Oct 31 03:18:56.025517 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.025490 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:18:56.306507 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.306307 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:18:56.912448 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.912169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:56.912448 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.912253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:56.912448 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.912265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:56.912448 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:56.912397 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:18:56.912448 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:56.912433 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:18:56.912800 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:56.912516 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:18:56.920470 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.920366 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:18:56.306329319Z","UUID":"e58bca9e-19f0-4460-abc3-0753b9eec4f0","Handler":null,"Name":"","Endpoint":""} Oct 31 03:18:56.922019 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.921998 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:18:56.922137 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:56.922033 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:18:57.029041 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:57.029002 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kh444" event={"ID":"c6c807d9-708e-4527-8e56-11749a26ca61","Type":"ContainerStarted","Data":"6044afd6614158ead82cac9737f54a6684ea4929c0c0085b7a800b706d39f7bf"} Oct 31 03:18:57.031521 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:57.031481 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" event={"ID":"3cfceaec-d546-4c9e-ac02-e66eeeea477a","Type":"ContainerStarted","Data":"5207a40fd9e08b7ceb2ccb047a3d154f186eb835b1c8005b9deaf0f057acc1bd"} Oct 31 03:18:57.034287 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:57.033698 2575 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="ad75098623277f700d60c008a37718a6da7cf0dda9e87a6773defbbe20dfb85f" exitCode=0 Oct 31 03:18:57.034287 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:57.033768 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerDied","Data":"ad75098623277f700d60c008a37718a6da7cf0dda9e87a6773defbbe20dfb85f"} Oct 31 03:18:57.036036 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:57.036016 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jw6nt" event={"ID":"93cfa49d-0426-4d23-ad84-50f12272a871","Type":"ContainerStarted","Data":"69d48209b43e30f9226113091fce981b0d2cb60a9ca8b6ee451c3e17d533680a"} Oct 31 03:18:57.270007 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:57.269912 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kh444" podStartSLOduration=3.670189733 podStartE2EDuration="12.269892556s" totalImagesPullingTime="8.599702823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:57.146422434 +0000 UTC m=+12.705925431" watchObservedRunningTime="2026-10-31 03:18:57.269892556 +0000 UTC m=+12.829395537" Oct 31 03:18:57.388677 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:57.388577 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jw6nt" podStartSLOduration=2.757377514 podStartE2EDuration="11.388559863s" totalImagesPullingTime="8.631182349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:57.388167506 +0000 UTC m=+12.947670475" watchObservedRunningTime="2026-10-31 03:18:57.388559863 +0000 UTC m=+12.948062829" Oct 31 03:18:58.040470 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:58.040436 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qgw8p" event={"ID":"bdd9600b-d5c9-4add-ae37-48aca3854510","Type":"ContainerStarted","Data":"3fcfaafb41cb160567806a6a5b41baa05bd4890d215fb138500ad5eba66a5153"} Oct 31 03:18:58.160040 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:58.159944 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qgw8p" podStartSLOduration=4.1828415230000004 podStartE2EDuration="13.159929745s" totalImagesPullingTime="8.977088222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:58.159494726 +0000 UTC m=+13.718997687" watchObservedRunningTime="2026-10-31 03:18:58.159929745 +0000 UTC m=+13.719432715" Oct 31 03:18:58.912754 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:58.912517 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:18:58.912883 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:58.912865 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:18:58.912948 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:58.912606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:18:58.912948 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:58.912522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:18:58.913050 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:58.912950 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:18:58.913105 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:18:58.913057 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:18:59.045306 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:59.045193 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" event={"ID":"3cfceaec-d546-4c9e-ac02-e66eeeea477a","Type":"ContainerStarted","Data":"8cc3128c365377a7d518410257de271fca916652e8902bc83e787cd50d92c94f"} Oct 31 03:18:59.047494 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:59.047456 2575 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="ee05b01eae1ac67eadc2b9a67d2ea2998ad6ee9133ac43d2edac2cf46504aa91" exitCode=0 Oct 31 03:18:59.047614 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:59.047547 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerDied","Data":"ee05b01eae1ac67eadc2b9a67d2ea2998ad6ee9133ac43d2edac2cf46504aa91"} Oct 31 03:18:59.049183 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:59.049156 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hhs74" event={"ID":"b39f2c1f-e94d-4a91-aabd-08b7094aaa38","Type":"ContainerStarted","Data":"e3e9f349cf744cb979436dd77bc8fb67bd6215ae02302f5509704d274b94e311"} Oct 31 03:18:59.165104 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:59.165051 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" podStartSLOduration=2.327087459 podStartE2EDuration="13.1650376s" totalImagesPullingTime="10.837950141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:59.164942256 +0000 UTC m=+14.724445225" watchObservedRunningTime="2026-10-31 03:18:59.1650376 +0000 UTC m=+14.724540566" Oct 31 03:18:59.419868 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:18:59.419779 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hhs74" podStartSLOduration=2.228517995 podStartE2EDuration="13.419764148s" totalImagesPullingTime="11.191246153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:59.419272959 +0000 UTC m=+14.978775930" watchObservedRunningTime="2026-10-31 03:18:59.419764148 +0000 UTC m=+14.979267116" Oct 31 03:19:00.913297 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:00.913194 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:00.913297 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:00.913249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:00.914123 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:00.913345 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:00.914123 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:00.913378 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:00.914123 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:00.913424 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:00.914123 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:00.913472 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:01.055746 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:01.055708 2575 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="ef6f1f4e35fd764356ee664b82dae187be33c4f9d0a0e7bdd5c504af7de676fc" exitCode=0 Oct 31 03:19:01.055918 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:01.055762 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerDied","Data":"ef6f1f4e35fd764356ee664b82dae187be33c4f9d0a0e7bdd5c504af7de676fc"} Oct 31 03:19:02.804431 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:02.804150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804371 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:02.804490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804513 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:02.804524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804532 2575 projected.go:196] Error preparing data for projected volume kube-api-access-25l6m for pod openshift-network-diagnostics/network-check-target-wdw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804597 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m podName:f15c9c88-bfe3-4acd-8047-9c3d9cd0296d nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.804576315 +0000 UTC m=+34.364079284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-25l6m" (UniqueName: "kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m") pod "network-check-target-wdw6h" (UID: "f15c9c88-bfe3-4acd-8047-9c3d9cd0296d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804643 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804644 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804701 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.804682998 +0000 UTC m=+34.364185960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:19:02.804868 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.804721 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret podName:3eefb160-a377-431f-be0a-c2bea6ecb352 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.804710246 +0000 UTC m=+34.364213204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret") pod "global-pull-secret-syncer-5ngpx" (UID: "3eefb160-a377-431f-be0a-c2bea6ecb352") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:19:02.912178 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:02.912009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:02.912178 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:02.912041 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:02.912178 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.912134 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:02.912386 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:02.912192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:02.912386 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.912303 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:02.912386 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:02.912379 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:03.064286 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:03.064254 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"559587c7864ae359fe176b32bef4385b0dd5ad590bffbdef43d1c62fad6b6d5a"} Oct 31 03:19:03.064452 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:03.064339 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"68fd51c219d57d6a61d28b36d1659eb5cbf4f9b812f4defae812fe3180509d9f"} Oct 31 03:19:03.064452 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:03.064398 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"8d8c2c132544755defa50f9e3ec05b75426d6cdb9d79091fee9e041acb4e5b1d"} Oct 31 03:19:03.064452 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:03.064440 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"ce693aa3c7d604bd37319a6c3f94090ee221729c28d8434ba658b6ffe7a82732"} Oct 31 03:19:03.064570 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:03.064473 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"adf815bcd1dbde2aa8dc9a89e232754d25bb156d1c62d0e2b8589c919ed3f364"} Oct 31 03:19:03.064570 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:03.064500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"798f8e1cfd7e1d3783a7bd0651233213b8675eea90f6f5f4a078f8759f1c34f5"} Oct 31 03:19:04.068855 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:04.068679 2575 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="8c6de07e13b9c354531cb9a954fcbaabe0757a7be68a0cb77ce347e12981e5da" exitCode=0 Oct 31 03:19:04.068855 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:04.068756 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerDied","Data":"8c6de07e13b9c354531cb9a954fcbaabe0757a7be68a0cb77ce347e12981e5da"} Oct 31 03:19:04.913546 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:04.913498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:04.913749 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:04.913595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:04.913749 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:04.913599 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:04.913749 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:04.913619 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:04.913749 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:04.913687 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:04.914001 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:04.913761 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:05.074846 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:05.074812 2575 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="5a8d9c7f7e41324bfb39823b52d39aa749e88a7dedf17087dfd372c074f6672a" exitCode=0 Oct 31 03:19:05.075370 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:05.074893 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerDied","Data":"5a8d9c7f7e41324bfb39823b52d39aa749e88a7dedf17087dfd372c074f6672a"} Oct 31 03:19:05.078726 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:05.078705 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"10af74136ac33f3dccbce7c071736e35a8c49d20aa611a25f5e89f375e348e02"} Oct 31 03:19:06.084422 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:06.084385 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" event={"ID":"1687ced2-8f63-4900-9b49-275a8104e159","Type":"ContainerStarted","Data":"58ecf9381ef6c8e078508e3fd196024ae59c49a53d3abf9c013fd826fa5b00b9"} Oct 31 03:19:06.207615 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:06.207562 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2r4ff" podStartSLOduration=6.732285103 podStartE2EDuration="20.207547931s" totalImagesPullingTime="13.475262828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:06.207319266 +0000 UTC m=+21.766822236" watchObservedRunningTime="2026-10-31 03:19:06.207547931 +0000 UTC m=+21.767050900" Oct 31 03:19:06.912784 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:06.912743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:06.912981 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:06.912743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:06.912981 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:06.912869 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:06.912981 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:06.912759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:06.913129 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:06.912920 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:06.913129 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:06.913008 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:08.092608 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:08.092570 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" event={"ID":"cf4f6953-b84a-482d-b685-bd2addcbafde","Type":"ContainerStarted","Data":"f458f5adcc7e1bca64edf80c7c1947ab004ead30caed6dc9b5c81c3f4fd1c81d"} Oct 31 03:19:08.093353 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:08.092840 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:08.109159 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:08.109134 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:08.373706 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:08.373590 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" podStartSLOduration=8.04495115 podStartE2EDuration="23.37357306s" totalImagesPullingTime="15.32862191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:08.373128204 +0000 UTC m=+23.932631174" watchObservedRunningTime="2026-10-31 03:19:08.37357306 +0000 UTC m=+23.933076031" Oct 31 03:19:08.912805 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:08.912770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:08.912991 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:08.912784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:08.912991 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:08.912939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:08.912991 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:08.912900 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:08.913143 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:08.912986 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:08.913143 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:08.913084 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:09.094553 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.094523 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:09.095030 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.095008 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:09.110787 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.110763 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:09.786547 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.786375 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ngpx"] Oct 31 03:19:09.786735 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.786720 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:09.786862 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:09.786840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:09.791796 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.791761 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wdw6h"] Oct 31 03:19:09.791932 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.791876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:09.792002 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:09.791976 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:09.792753 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.792731 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2hw8"] Oct 31 03:19:09.792836 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:09.792830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:09.792934 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:09.792916 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:10.096633 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:10.096541 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:11.098823 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:11.098793 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:11.167672 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:11.167631 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:19:11.167850 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:11.167797 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:11.168200 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:11.168177 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:19:11.912282 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:11.912246 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:11.912461 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:11.912249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:11.912461 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:11.912361 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:11.912461 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:11.912249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:11.912461 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:11.912415 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:11.912622 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:11.912501 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:13.636847 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:13.636811 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:13.637242 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:13.637088 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:13.654479 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:13.654450 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:13.912449 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:13.912367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:13.912588 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:13.912476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:13.912588 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:13.912483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:13.912588 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:13.912566 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wdw6h" podUID="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" Oct 31 03:19:13.912754 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:13.912481 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ngpx" podUID="3eefb160-a377-431f-be0a-c2bea6ecb352" Oct 31 03:19:13.912754 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:13.912636 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2hw8" podUID="9ec0a926-6081-4a94-916a-2e27400b06a9" Oct 31 03:19:14.748814 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.748784 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeReady" Oct 31 03:19:14.749161 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.748924 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 03:19:14.864398 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.864358 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mdl5l"] Oct 31 03:19:14.886847 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.886809 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mdl5v"] Oct 31 03:19:14.887004 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.886934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" Oct 31 03:19:14.898673 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.898646 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xbfpv"] Oct 31 03:19:14.898794 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.898750 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" Oct 31 03:19:14.908722 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.908700 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx"] Oct 31 03:19:14.908830 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.908804 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:14.919349 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.919323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:14.927599 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.927574 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww"] Oct 31 03:19:14.939991 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.939962 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkgt6"] Oct 31 03:19:14.940154 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.940096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:14.946545 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.946520 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d69c76ccf-dccpx"] Oct 31 03:19:14.946670 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.946639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" Oct 31 03:19:14.953184 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.953145 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zrtsh"] Oct 31 03:19:14.953321 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.953269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:14.958346 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.958319 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7"] Oct 31 03:19:14.958458 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.958409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:14.961047 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.961024 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks"] Oct 31 03:19:14.961467 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.961448 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:14.964464 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.964441 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h"] Oct 31 03:19:14.964701 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.964684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" Oct 31 03:19:14.966498 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.966475 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 03:19:14.966589 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.966525 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 03:19:14.966713 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.966693 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 03:19:14.966882 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.966822 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 03:19:14.966956 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.966914 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rnwzl\"" Oct 31 03:19:14.971063 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.971036 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-b54xh"] Oct 31 03:19:14.971158 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.971143 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:14.971878 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.971860 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 03:19:14.972432 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.972418 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 03:19:14.978965 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.978906 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79f9f4c5f5-j4qrr"] Oct 31 03:19:14.979346 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.979329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:14.989680 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.989655 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58d7548854-gr79z"] Oct 31 03:19:14.989804 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.989784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:14.990352 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.990153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/15c8c5bb-10f1-4894-b45d-34b2209f9676-kube-api-access-nnfr7\") pod \"kas-connection-checker-685895c7c9-mdl5l\" (UID: \"15c8c5bb-10f1-4894-b45d-34b2209f9676\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" Oct 31 03:19:14.990352 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.990314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtgd\" (UniqueName: \"kubernetes.io/projected/69dbc1e2-7632-40e1-8e81-79cf08b42d67-kube-api-access-jxtgd\") pod \"kas-connection-checker-685895c7c9-mdl5v\" (UID: \"69dbc1e2-7632-40e1-8e81-79cf08b42d67\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" Oct 31 03:19:14.992112 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.992090 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw"] Oct 31 03:19:14.992326 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:14.992265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.003368 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.003341 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4k5qr"] Oct 31 03:19:15.003540 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.003522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" Oct 31 03:19:15.014441 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.014416 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pvphc"] Oct 31 03:19:15.014571 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.014554 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.021558 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021529 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mdl5l"] Oct 31 03:19:15.021558 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021561 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xbfpv"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021576 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021594 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mdl5v"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021608 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021621 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021639 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021660 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-b54xh"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021683 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f9f4c5f5-j4qrr"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021703 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d7548854-gr79z"] Oct 31 03:19:15.021716 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.021658 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.032250 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.030693 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks"] Oct 31 03:19:15.032250 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.030759 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pvphc"] Oct 31 03:19:15.035713 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.035683 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4k5qr"] Oct 31 03:19:15.035864 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.035751 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkgt6"] Oct 31 03:19:15.035864 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.035777 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zrtsh"] Oct 31 03:19:15.037562 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.037541 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d69c76ccf-dccpx"] Oct 31 03:19:15.045383 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.044997 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw"] Oct 31 03:19:15.046710 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.046683 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 03:19:15.047295 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.047271 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.047369 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.047322 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.047429 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.047396 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.047555 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.047463 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.047725 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.047630 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 03:19:15.047832 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.047731 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.048481 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048464 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.048574 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048508 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.048574 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048515 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 03:19:15.048574 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048553 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.048574 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048570 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.048779 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048582 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.048779 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048611 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 03:19:15.048779 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048635 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:19:15.048779 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048668 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.048779 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048667 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.048990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048794 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 03:19:15.048990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048804 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.048990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048836 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.048990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048878 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.048990 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.048894 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.091247 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n2t5\" (UniqueName: \"kubernetes.io/projected/3aed8bd2-021f-4e33-96f0-e9c974fa0701-kube-api-access-2n2t5\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.091414 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a57d414f-4fb4-4b78-92c7-105e2ac19d21-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.091414 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.091484 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.091484 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtgd\" (UniqueName: \"kubernetes.io/projected/69dbc1e2-7632-40e1-8e81-79cf08b42d67-kube-api-access-jxtgd\") pod \"kas-connection-checker-685895c7c9-mdl5v\" (UID: \"69dbc1e2-7632-40e1-8e81-79cf08b42d67\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" Oct 31 03:19:15.091551 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.091590 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.091590 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zth\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.091724 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27js\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.091792 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aed8bd2-021f-4e33-96f0-e9c974fa0701-config\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.091859 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ww68\" (UniqueName: \"kubernetes.io/projected/3a5ed6d1-a843-4d21-b913-59b0f9221e42-kube-api-access-8ww68\") pod \"volume-data-source-validator-58c9f5f46d-87lks\" (UID: \"3a5ed6d1-a843-4d21-b913-59b0f9221e42\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" Oct 31 03:19:15.091859 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjmw\" (UniqueName: \"kubernetes.io/projected/283b8014-5ffd-4671-a28d-2975376d9cf9-kube-api-access-kkjmw\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.091917 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.091944 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.091985 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.092023 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.091996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aed8bd2-021f-4e33-96f0-e9c974fa0701-serving-cert\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.092055 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/15c8c5bb-10f1-4894-b45d-34b2209f9676-kube-api-access-nnfr7\") pod \"kas-connection-checker-685895c7c9-mdl5l\" (UID: \"15c8c5bb-10f1-4894-b45d-34b2209f9676\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" Oct 31 03:19:15.092055 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-default-certificate\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.092128 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a57d414f-4fb4-4b78-92c7-105e2ac19d21-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.092244 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqpm8\" (UniqueName: \"kubernetes.io/projected/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-kube-api-access-zqpm8\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.092292 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.092340 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.092381 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsgb\" (UniqueName: \"kubernetes.io/projected/a57d414f-4fb4-4b78-92c7-105e2ac19d21-kube-api-access-xfsgb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.092381 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-trusted-ca\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.092452 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlc5\" (UniqueName: \"kubernetes.io/projected/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-kube-api-access-xvlc5\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.092452 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c3675be1-c31d-423f-83df-295df033f31a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:15.092514 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-snapshots\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.092545 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.092579 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2ps\" (UniqueName: \"kubernetes.io/projected/9f6df5b8-3430-46ee-8af1-13bb15bda816-kube-api-access-xs2ps\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.092619 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xtp\" (UniqueName: \"kubernetes.io/projected/2901d41f-f26c-4e6b-909f-1fc449e11ce8-kube-api-access-s8xtp\") pod \"network-check-source-6f78d59f99-t4nxw\" (UID: \"2901d41f-f26c-4e6b-909f-1fc449e11ce8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" Oct 31 03:19:15.092619 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jgl\" (UniqueName: \"kubernetes.io/projected/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6-kube-api-access-w8jgl\") pod \"kas-connection-checker-685895c7c9-pkgt6\" (UID: \"4cc36af9-eaa8-40a3-b642-cc7aaf970ed6\") " pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" Oct 31 03:19:15.092684 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmczk\" (UniqueName: \"kubernetes.io/projected/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-kube-api-access-jmczk\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.092684 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.092744 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.092744 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-serving-cert\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.092744 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-tmp\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.092848 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6df5b8-3430-46ee-8af1-13bb15bda816-serving-cert\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.092848 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-service-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.092848 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.092944 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-stats-auth\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.092944 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.093009 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.092988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/283b8014-5ffd-4671-a28d-2975376d9cf9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.093042 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.093080 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-config\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.093112 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.093159 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:15.093195 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.093262 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.093262 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.093349 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.093333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.099322 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099283 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 03:19:15.099461 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099359 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 03:19:15.099461 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099295 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 03:19:15.099461 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099357 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 03:19:15.099596 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099471 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jjf7q\"" Oct 31 03:19:15.099596 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099524 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:19:15.099596 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099552 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7qxg9\"" Oct 31 03:19:15.099748 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099612 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 03:19:15.099748 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099628 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 03:19:15.099748 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099662 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hhcvs\"" Oct 31 03:19:15.099748 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099703 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tgrsk\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099753 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099800 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099814 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8zl9c\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099847 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099869 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099873 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099755 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mnxkk\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099923 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 03:19:15.099963 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099944 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-n6jvn\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.099992 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100027 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-242zg\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100065 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btzr7\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100083 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100148 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100161 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-q872x\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100222 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fn2lp\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100290 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 03:19:15.100380 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100338 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:19:15.100807 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100683 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 03:19:15.100807 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100804 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:19:15.100900 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.100874 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 03:19:15.155083 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.155007 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/15c8c5bb-10f1-4894-b45d-34b2209f9676-kube-api-access-nnfr7\") pod \"kas-connection-checker-685895c7c9-mdl5l\" (UID: \"15c8c5bb-10f1-4894-b45d-34b2209f9676\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" Oct 31 03:19:15.155083 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.155051 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtgd\" (UniqueName: \"kubernetes.io/projected/69dbc1e2-7632-40e1-8e81-79cf08b42d67-kube-api-access-jxtgd\") pod \"kas-connection-checker-685895c7c9-mdl5v\" (UID: \"69dbc1e2-7632-40e1-8e81-79cf08b42d67\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" Oct 31 03:19:15.194167 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.193976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27js\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.194314 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aed8bd2-021f-4e33-96f0-e9c974fa0701-config\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.194314 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww68\" (UniqueName: \"kubernetes.io/projected/3a5ed6d1-a843-4d21-b913-59b0f9221e42-kube-api-access-8ww68\") pod \"volume-data-source-validator-58c9f5f46d-87lks\" (UID: \"3a5ed6d1-a843-4d21-b913-59b0f9221e42\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" Oct 31 03:19:15.194314 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjmw\" (UniqueName: \"kubernetes.io/projected/283b8014-5ffd-4671-a28d-2975376d9cf9-kube-api-access-kkjmw\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.194314 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.194314 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.194372 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.194390 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5fbb4c-6297-4655-8c67-ff336a28e948-tmp-dir\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.194442 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.694425349 +0000 UTC m=+31.253928297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aed8bd2-021f-4e33-96f0-e9c974fa0701-serving-cert\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-default-certificate\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.194585 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.194582 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a57d414f-4fb4-4b78-92c7-105e2ac19d21-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.194636 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.694623911 +0000 UTC m=+31.254126859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpm8\" (UniqueName: \"kubernetes.io/projected/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-kube-api-access-zqpm8\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsgb\" (UniqueName: \"kubernetes.io/projected/a57d414f-4fb4-4b78-92c7-105e2ac19d21-kube-api-access-xfsgb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-trusted-ca\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlc5\" (UniqueName: \"kubernetes.io/projected/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-kube-api-access-xvlc5\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c3675be1-c31d-423f-83df-295df033f31a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-snapshots\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2ps\" (UniqueName: \"kubernetes.io/projected/9f6df5b8-3430-46ee-8af1-13bb15bda816-kube-api-access-xs2ps\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xtp\" (UniqueName: \"kubernetes.io/projected/2901d41f-f26c-4e6b-909f-1fc449e11ce8-kube-api-access-s8xtp\") pod \"network-check-source-6f78d59f99-t4nxw\" (UID: \"2901d41f-f26c-4e6b-909f-1fc449e11ce8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" Oct 31 03:19:15.195010 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194957 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.194985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jgl\" (UniqueName: \"kubernetes.io/projected/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6-kube-api-access-w8jgl\") pod \"kas-connection-checker-685895c7c9-pkgt6\" (UID: \"4cc36af9-eaa8-40a3-b642-cc7aaf970ed6\") " pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmczk\" (UniqueName: \"kubernetes.io/projected/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-kube-api-access-jmczk\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-serving-cert\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195273 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a57d414f-4fb4-4b78-92c7-105e2ac19d21-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-tmp\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6df5b8-3430-46ee-8af1-13bb15bda816-serving-cert\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.195769 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-service-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xj84\" (UniqueName: \"kubernetes.io/projected/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-kube-api-access-5xj84\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fbb4c-6297-4655-8c67-ff336a28e948-config-volume\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-stats-auth\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/283b8014-5ffd-4671-a28d-2975376d9cf9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.195990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zsl\" (UniqueName: \"kubernetes.io/projected/1c5fbb4c-6297-4655-8c67-ff336a28e948-kube-api-access-74zsl\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-config\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-snapshots\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.196160 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196198 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-trusted-ca\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2t5\" (UniqueName: \"kubernetes.io/projected/3aed8bd2-021f-4e33-96f0-e9c974fa0701-kube-api-access-2n2t5\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a57d414f-4fb4-4b78-92c7-105e2ac19d21-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-cert\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zth\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.196884 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.196683 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.197486 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.197044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c3675be1-c31d-423f-83df-295df033f31a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:15.197486 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.197135 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:15.197486 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.197152 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:15.197486 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.197214 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.69719604 +0000 UTC m=+31.256699001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:15.197865 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.197835 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.197982 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.197958 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.198221 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.198016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aed8bd2-021f-4e33-96f0-e9c974fa0701-config\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.198307 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.198293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/283b8014-5ffd-4671-a28d-2975376d9cf9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.198404 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.198395 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:15.198459 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.198441 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.698426243 +0000 UTC m=+31.257929205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:15.198527 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.198461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.198648 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.198623 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.198761 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.198740 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-serving-cert\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.199042 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.199017 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-tmp\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.199108 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.199063 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.199108 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.199069 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-config\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.199208 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.199118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-service-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.199208 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.199159 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:15.199208 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.199202 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.699187738 +0000 UTC m=+31.258690705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:15.199386 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.199299 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.69928795 +0000 UTC m=+31.258790926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:15.199447 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.199410 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.199515 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.199495 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:15.199567 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.199550 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.699535212 +0000 UTC m=+31.259038166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:15.200084 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.200056 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.200222 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.200196 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a57d414f-4fb4-4b78-92c7-105e2ac19d21-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.200403 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.200374 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-default-certificate\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.201676 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.201649 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.201676 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.201644 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.201840 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.201820 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.202026 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.202004 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-stats-auth\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.206528 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.202292 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aed8bd2-021f-4e33-96f0-e9c974fa0701-serving-cert\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.206528 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.202779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6df5b8-3430-46ee-8af1-13bb15bda816-serving-cert\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.206967 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.206941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" Oct 31 03:19:15.216921 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.216894 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" Oct 31 03:19:15.258711 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.257756 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjmw\" (UniqueName: \"kubernetes.io/projected/283b8014-5ffd-4671-a28d-2975376d9cf9-kube-api-access-kkjmw\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.258711 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.257775 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27js\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.258711 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.257890 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpm8\" (UniqueName: \"kubernetes.io/projected/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-kube-api-access-zqpm8\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.258711 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.258293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsgb\" (UniqueName: \"kubernetes.io/projected/a57d414f-4fb4-4b78-92c7-105e2ac19d21-kube-api-access-xfsgb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.258711 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.258342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww68\" (UniqueName: \"kubernetes.io/projected/3a5ed6d1-a843-4d21-b913-59b0f9221e42-kube-api-access-8ww68\") pod \"volume-data-source-validator-58c9f5f46d-87lks\" (UID: \"3a5ed6d1-a843-4d21-b913-59b0f9221e42\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" Oct 31 03:19:15.259090 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.259020 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlc5\" (UniqueName: \"kubernetes.io/projected/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-kube-api-access-xvlc5\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.259837 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.259749 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xtp\" (UniqueName: \"kubernetes.io/projected/2901d41f-f26c-4e6b-909f-1fc449e11ce8-kube-api-access-s8xtp\") pod \"network-check-source-6f78d59f99-t4nxw\" (UID: \"2901d41f-f26c-4e6b-909f-1fc449e11ce8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" Oct 31 03:19:15.260726 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.260707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmczk\" (UniqueName: \"kubernetes.io/projected/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-kube-api-access-jmczk\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.261012 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.260990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.261496 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.261473 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2t5\" (UniqueName: \"kubernetes.io/projected/3aed8bd2-021f-4e33-96f0-e9c974fa0701-kube-api-access-2n2t5\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.261624 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.261608 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2ps\" (UniqueName: \"kubernetes.io/projected/9f6df5b8-3430-46ee-8af1-13bb15bda816-kube-api-access-xs2ps\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.262101 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.262061 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jgl\" (UniqueName: \"kubernetes.io/projected/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6-kube-api-access-w8jgl\") pod \"kas-connection-checker-685895c7c9-pkgt6\" (UID: \"4cc36af9-eaa8-40a3-b642-cc7aaf970ed6\") " pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" Oct 31 03:19:15.262172 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.262161 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zth\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.263322 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.263291 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.280458 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.280425 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:15.297874 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.297834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xj84\" (UniqueName: \"kubernetes.io/projected/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-kube-api-access-5xj84\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.298037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.297883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fbb4c-6297-4655-8c67-ff336a28e948-config-volume\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.298037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.297927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zsl\" (UniqueName: \"kubernetes.io/projected/1c5fbb4c-6297-4655-8c67-ff336a28e948-kube-api-access-74zsl\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.298037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.298011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-cert\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.298219 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.298079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5fbb4c-6297-4655-8c67-ff336a28e948-tmp-dir\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.298690 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.298503 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5fbb4c-6297-4655-8c67-ff336a28e948-tmp-dir\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.298690 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.298586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.299068 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.299038 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fbb4c-6297-4655-8c67-ff336a28e948-config-volume\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.299336 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.299319 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:15.299398 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.299377 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:15.799364068 +0000 UTC m=+31.358867017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:15.302649 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.302621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-cert\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.307509 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.307488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" Oct 31 03:19:15.317970 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.317916 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:15.332728 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.332700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:15.344761 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.344478 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" Oct 31 03:19:15.373246 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.373202 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xj84\" (UniqueName: \"kubernetes.io/projected/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-kube-api-access-5xj84\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.382743 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.382707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zsl\" (UniqueName: \"kubernetes.io/projected/1c5fbb4c-6297-4655-8c67-ff336a28e948-kube-api-access-74zsl\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.554181 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.554145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:15.560290 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.560264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" Oct 31 03:19:15.659046 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.658814 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mdl5l"] Oct 31 03:19:15.659588 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.659572 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:15.661732 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:15.661709 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c8c5bb_10f1_4894_b45d_34b2209f9676.slice/crio-38b5a4b1d9e003591d385e70a51f3306a15cae63ce0f9be186bced90abe010f3 WatchSource:0}: Error finding container 38b5a4b1d9e003591d385e70a51f3306a15cae63ce0f9be186bced90abe010f3: Status 404 returned error can't find the container with id 38b5a4b1d9e003591d385e70a51f3306a15cae63ce0f9be186bced90abe010f3 Oct 31 03:19:15.702858 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.702673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:15.702858 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.702729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:15.702974 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.702853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:15.702974 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.702889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.702974 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.702917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:15.702974 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.702942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:15.702974 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.702971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:15.703156 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703135 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:15.703156 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703158 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703219 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.703200572 +0000 UTC m=+32.262703542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703262 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703289 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703293 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703313 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703316 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.703303968 +0000 UTC m=+32.262806931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703338 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.703326723 +0000 UTC m=+32.262829673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:15.703345 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703353 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.703344614 +0000 UTC m=+32.262847566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:15.703637 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703362 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:15.703637 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703370 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.703361568 +0000 UTC m=+32.262864517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:15.703637 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703382 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:15.703637 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703391 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.703379015 +0000 UTC m=+32.262881964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:15.703637 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.703420 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.703405485 +0000 UTC m=+32.262908448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:15.710177 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.710151 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zrtsh"] Oct 31 03:19:15.713288 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:15.713255 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6df5b8_3430_46ee_8af1_13bb15bda816.slice/crio-a1e1ea2182e46ce3b256ed3cc00bb75a1345a7cab033c2ce5cb8689a76cc3b10 WatchSource:0}: Error finding container a1e1ea2182e46ce3b256ed3cc00bb75a1345a7cab033c2ce5cb8689a76cc3b10: Status 404 returned error can't find the container with id a1e1ea2182e46ce3b256ed3cc00bb75a1345a7cab033c2ce5cb8689a76cc3b10 Oct 31 03:19:15.724265 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.723928 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks"] Oct 31 03:19:15.759267 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.754992 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h"] Oct 31 03:19:15.774468 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.774449 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-b54xh"] Oct 31 03:19:15.776620 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:15.776592 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdf0731_bef7_4c9c_a5c3_09eff5aef394.slice/crio-7c3b90117d898e769619cdec74f18d1e69f25206f4b6d63d100b8c6e5dd29e59 WatchSource:0}: Error finding container 7c3b90117d898e769619cdec74f18d1e69f25206f4b6d63d100b8c6e5dd29e59: Status 404 returned error can't find the container with id 7c3b90117d898e769619cdec74f18d1e69f25206f4b6d63d100b8c6e5dd29e59 Oct 31 03:19:15.777128 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.777058 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw"] Oct 31 03:19:15.780042 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.780022 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mdl5v"] Oct 31 03:19:15.780420 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:15.780350 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2901d41f_f26c_4e6b_909f_1fc449e11ce8.slice/crio-7befa522b24d9bfa6b0232bb97c32b94861ad61222826e8fa39ad6abc0dd4807 WatchSource:0}: Error finding container 7befa522b24d9bfa6b0232bb97c32b94861ad61222826e8fa39ad6abc0dd4807: Status 404 returned error can't find the container with id 7befa522b24d9bfa6b0232bb97c32b94861ad61222826e8fa39ad6abc0dd4807 Oct 31 03:19:15.782665 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:15.782643 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dbc1e2_7632_40e1_8e81_79cf08b42d67.slice/crio-8bffa4ab83c06e166284c2b70616242842d5edae65e73d11961aefbfdd1cf7c7 WatchSource:0}: Error finding container 8bffa4ab83c06e166284c2b70616242842d5edae65e73d11961aefbfdd1cf7c7: Status 404 returned error can't find the container with id 8bffa4ab83c06e166284c2b70616242842d5edae65e73d11961aefbfdd1cf7c7 Oct 31 03:19:15.803861 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.803829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:15.804051 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.804029 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:15.804160 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:15.804094 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:16.804076123 +0000 UTC m=+32.363579079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:15.912744 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.912660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:15.912859 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.912754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:15.912859 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.912763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:15.938838 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.938811 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww"] Oct 31 03:19:15.942520 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:15.942495 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aed8bd2_021f_4e33_96f0_e9c974fa0701.slice/crio-3b1ba02d33e619ce2f9882279d307b46171da0e12c021ffce8557a14b78916a6 WatchSource:0}: Error finding container 3b1ba02d33e619ce2f9882279d307b46171da0e12c021ffce8557a14b78916a6: Status 404 returned error can't find the container with id 3b1ba02d33e619ce2f9882279d307b46171da0e12c021ffce8557a14b78916a6 Oct 31 03:19:15.954342 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.954263 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkgt6"] Oct 31 03:19:15.956745 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:15.956720 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc36af9_eaa8_40a3_b642_cc7aaf970ed6.slice/crio-52b0684eb07236bb6cbe33e42f936e5aead498ee528359dc2d37bf332b6d746d WatchSource:0}: Error finding container 52b0684eb07236bb6cbe33e42f936e5aead498ee528359dc2d37bf332b6d746d: Status 404 returned error can't find the container with id 52b0684eb07236bb6cbe33e42f936e5aead498ee528359dc2d37bf332b6d746d Oct 31 03:19:15.966470 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.966434 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:19:15.966470 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.966441 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkxjs\"" Oct 31 03:19:15.966619 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.966500 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:19:15.966619 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.966500 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vxflf\"" Oct 31 03:19:15.966619 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:15.966574 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v5ddh\"" Oct 31 03:19:16.072440 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.072415 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pvphc"] Oct 31 03:19:16.075413 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:16.075386 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abc9860_5a1c_4ca0_a8ab_ee18d0a21b90.slice/crio-fe6c8824f78c4b17885dfe6daaf9774b4f1bc7a68260cb6bfed3971a24e588dc WatchSource:0}: Error finding container fe6c8824f78c4b17885dfe6daaf9774b4f1bc7a68260cb6bfed3971a24e588dc: Status 404 returned error can't find the container with id fe6c8824f78c4b17885dfe6daaf9774b4f1bc7a68260cb6bfed3971a24e588dc Oct 31 03:19:16.110655 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.110580 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zrtsh" event={"ID":"9f6df5b8-3430-46ee-8af1-13bb15bda816","Type":"ContainerStarted","Data":"a1e1ea2182e46ce3b256ed3cc00bb75a1345a7cab033c2ce5cb8689a76cc3b10"} Oct 31 03:19:16.112042 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.112018 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" event={"ID":"69dbc1e2-7632-40e1-8e81-79cf08b42d67","Type":"ContainerStarted","Data":"afd7f519932b2218818c422ac451de2653c1491bc3b618bdc569261b4895e8bd"} Oct 31 03:19:16.112147 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.112068 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" event={"ID":"69dbc1e2-7632-40e1-8e81-79cf08b42d67","Type":"ContainerStarted","Data":"8bffa4ab83c06e166284c2b70616242842d5edae65e73d11961aefbfdd1cf7c7"} Oct 31 03:19:16.113337 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.113310 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" event={"ID":"a57d414f-4fb4-4b78-92c7-105e2ac19d21","Type":"ContainerStarted","Data":"f5419e24989d40d3a07f438241f3aab44261dc54a0f4a379a2a1c2e63daa8fa1"} Oct 31 03:19:16.114555 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.114533 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" event={"ID":"3a5ed6d1-a843-4d21-b913-59b0f9221e42","Type":"ContainerStarted","Data":"d2e6ccf3cf471d62cd75c792e3ef1f94f53c4af0f747896364dcf8a8d962f6e2"} Oct 31 03:19:16.115631 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.115601 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" event={"ID":"3aed8bd2-021f-4e33-96f0-e9c974fa0701","Type":"ContainerStarted","Data":"3b1ba02d33e619ce2f9882279d307b46171da0e12c021ffce8557a14b78916a6"} Oct 31 03:19:16.117021 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.117003 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" event={"ID":"4cc36af9-eaa8-40a3-b642-cc7aaf970ed6","Type":"ContainerStarted","Data":"c5571bd65c25ec929a1bdeceefed0d6ba7b399cad729682c43317a44f81b86e0"} Oct 31 03:19:16.117108 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.117035 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" event={"ID":"4cc36af9-eaa8-40a3-b642-cc7aaf970ed6","Type":"ContainerStarted","Data":"52b0684eb07236bb6cbe33e42f936e5aead498ee528359dc2d37bf332b6d746d"} Oct 31 03:19:16.118309 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.118290 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" event={"ID":"2901d41f-f26c-4e6b-909f-1fc449e11ce8","Type":"ContainerStarted","Data":"7befa522b24d9bfa6b0232bb97c32b94861ad61222826e8fa39ad6abc0dd4807"} Oct 31 03:19:16.119412 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.119382 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" event={"ID":"9bdf0731-bef7-4c9c-a5c3-09eff5aef394","Type":"ContainerStarted","Data":"7c3b90117d898e769619cdec74f18d1e69f25206f4b6d63d100b8c6e5dd29e59"} Oct 31 03:19:16.120796 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.120774 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" event={"ID":"15c8c5bb-10f1-4894-b45d-34b2209f9676","Type":"ContainerStarted","Data":"b10a573b4195aa912881a5024cbfd430875249688d940a3b15511b769738a981"} Oct 31 03:19:16.120899 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.120808 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" event={"ID":"15c8c5bb-10f1-4894-b45d-34b2209f9676","Type":"ContainerStarted","Data":"38b5a4b1d9e003591d385e70a51f3306a15cae63ce0f9be186bced90abe010f3"} Oct 31 03:19:16.121857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.121840 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pvphc" event={"ID":"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90","Type":"ContainerStarted","Data":"fe6c8824f78c4b17885dfe6daaf9774b4f1bc7a68260cb6bfed3971a24e588dc"} Oct 31 03:19:16.717752 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.717710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:16.717919 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.717772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:16.717919 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.717867 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:16.717919 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.717901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.717929 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.717911508 +0000 UTC m=+34.277414476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.717959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.717969 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.717990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718026 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.718008513 +0000 UTC m=+34.277511478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718073 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718086 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:16.718100 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.718095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718121 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.718109444 +0000 UTC m=+34.277612427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.718149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718164 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718176 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718214 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.718199669 +0000 UTC m=+34.277702632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.717870 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718271 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718283 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.718272643 +0000 UTC m=+34.277775610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718307 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.718296141 +0000 UTC m=+34.277799107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:16.718462 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.718351 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.718339643 +0000 UTC m=+34.277842611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:16.819361 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:16.819316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:16.819780 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.819472 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:16.819780 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:16.819540 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:18.819521471 +0000 UTC m=+34.379024419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:17.060109 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:17.059887 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" podStartSLOduration=352.059871038 podStartE2EDuration="5m52.059871038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:17.059241708 +0000 UTC m=+32.618744670" watchObservedRunningTime="2026-10-31 03:19:17.059871038 +0000 UTC m=+32.619374002" Oct 31 03:19:17.178192 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:17.178125 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" podStartSLOduration=352.178107598 podStartE2EDuration="5m52.178107598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:17.177087797 +0000 UTC m=+32.736590768" watchObservedRunningTime="2026-10-31 03:19:17.178107598 +0000 UTC m=+32.737610569" Oct 31 03:19:17.297202 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:17.297145 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" podStartSLOduration=352.297133552 podStartE2EDuration="5m52.297133552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:17.294306453 +0000 UTC m=+32.853809425" watchObservedRunningTime="2026-10-31 03:19:17.297133552 +0000 UTC m=+32.856636516" Oct 31 03:19:18.743156 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.743106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:18.743696 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.743162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:18.743696 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.743193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:18.743696 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.743349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:18.743696 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.743422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:18.743696 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.743460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:18.743696 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.743582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:18.744076 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.743721 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:18.744076 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.743787 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.743767422 +0000 UTC m=+38.303270377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:18.744076 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.743850 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:18.744076 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.743884 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.743871624 +0000 UTC m=+38.303374586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:18.744076 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.743935 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:18.744076 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.743966 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.743955846 +0000 UTC m=+38.303458800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:18.744076 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744024 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.744014721 +0000 UTC m=+38.303517684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744098 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744109 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744139 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.744128946 +0000 UTC m=+38.303631898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744198 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744207 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744252 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.744242003 +0000 UTC m=+38.303744965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744312 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:18.744603 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.744341 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.74433221 +0000 UTC m=+38.303835177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:18.844287 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.844251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:18.844468 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.844308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:18.844468 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.844332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:18.844468 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.844366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:18.844468 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.844427 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:18.844662 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.844509 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.844490608 +0000 UTC m=+38.403993556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:18.844662 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.844626 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:19:18.844662 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:18.844651 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:50.844642663 +0000 UTC m=+66.404145610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : secret "metrics-daemon-secret" not found Oct 31 03:19:18.854273 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.854199 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:18.855694 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.855665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:18.937118 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.936923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:18.954123 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:18.954031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:19.144148 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:19.144066 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" event={"ID":"3a5ed6d1-a843-4d21-b913-59b0f9221e42","Type":"ContainerStarted","Data":"fa3696b8765b3d855256d11ceedac554488359e913072ea7e48638ba0e5d4ed6"} Oct 31 03:19:19.374146 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:19.374074 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" podStartSLOduration=318.562545283 podStartE2EDuration="5m21.374047033s" totalImagesPullingTime="2.81150175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:19.262477467 +0000 UTC m=+34.821980438" watchObservedRunningTime="2026-10-31 03:19:19.374047033 +0000 UTC m=+34.933550004" Oct 31 03:19:19.374516 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:19.374415 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ngpx"] Oct 31 03:19:19.382734 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:19.382702 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eefb160_a377_431f_be0a_c2bea6ecb352.slice/crio-19a9a3b840f5998529e86e283fa2ea71831b50665e81d68008d91f0b4897fca9 WatchSource:0}: Error finding container 19a9a3b840f5998529e86e283fa2ea71831b50665e81d68008d91f0b4897fca9: Status 404 returned error can't find the container with id 19a9a3b840f5998529e86e283fa2ea71831b50665e81d68008d91f0b4897fca9 Oct 31 03:19:19.432888 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:19.432860 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wdw6h"] Oct 31 03:19:19.435928 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:19.435895 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15c9c88_bfe3_4acd_8047_9c3d9cd0296d.slice/crio-86348cd6e950a49f812da1f9212057a5cc992ab5d5cbc68b6bf780a54031b307 WatchSource:0}: Error finding container 86348cd6e950a49f812da1f9212057a5cc992ab5d5cbc68b6bf780a54031b307: Status 404 returned error can't find the container with id 86348cd6e950a49f812da1f9212057a5cc992ab5d5cbc68b6bf780a54031b307 Oct 31 03:19:20.149664 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:20.149623 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wdw6h" event={"ID":"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d","Type":"ContainerStarted","Data":"86348cd6e950a49f812da1f9212057a5cc992ab5d5cbc68b6bf780a54031b307"} Oct 31 03:19:20.151479 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:20.151444 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5ngpx" event={"ID":"3eefb160-a377-431f-be0a-c2bea6ecb352","Type":"ContainerStarted","Data":"19a9a3b840f5998529e86e283fa2ea71831b50665e81d68008d91f0b4897fca9"} Oct 31 03:19:21.167602 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:21.167512 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" event={"ID":"9bdf0731-bef7-4c9c-a5c3-09eff5aef394","Type":"ContainerStarted","Data":"b0df75062e850088f77634f43a4abb72d7f3c30b82a8fa881e6cb58031454bb1"} Oct 31 03:19:21.170428 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:21.169509 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zrtsh" event={"ID":"9f6df5b8-3430-46ee-8af1-13bb15bda816","Type":"ContainerStarted","Data":"4f81404d77f91f09c2fdfc818650a1411e025c066b241095194a1ef15f7b866c"} Oct 31 03:19:22.179038 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.178452 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" event={"ID":"a57d414f-4fb4-4b78-92c7-105e2ac19d21","Type":"ContainerStarted","Data":"942be70ca7fc3135e157e052f3a5ced4f43e9f237f3b9cef2d6acf547cd9e822"} Oct 31 03:19:22.182897 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.182836 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" event={"ID":"3aed8bd2-021f-4e33-96f0-e9c974fa0701","Type":"ContainerStarted","Data":"e1a278165148cb6279a2c2bcb41488f8ff213cd2221a452ee6a04fb5c6819599"} Oct 31 03:19:22.187061 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.187036 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/0.log" Oct 31 03:19:22.187178 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.187077 2575 generic.go:356] "Generic (PLEG): container finished" podID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" containerID="b0df75062e850088f77634f43a4abb72d7f3c30b82a8fa881e6cb58031454bb1" exitCode=255 Oct 31 03:19:22.187309 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.187172 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" event={"ID":"9bdf0731-bef7-4c9c-a5c3-09eff5aef394","Type":"ContainerDied","Data":"b0df75062e850088f77634f43a4abb72d7f3c30b82a8fa881e6cb58031454bb1"} Oct 31 03:19:22.187357 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.187340 2575 scope.go:122] "RemoveContainer" containerID="b0df75062e850088f77634f43a4abb72d7f3c30b82a8fa881e6cb58031454bb1" Oct 31 03:19:22.189560 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.189537 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pvphc" event={"ID":"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90","Type":"ContainerStarted","Data":"b08c7a9a28881b0520f3408cbf9c1c7d5bf5f61d35a30242296e61847f147281"} Oct 31 03:19:22.428970 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.428913 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" podStartSLOduration=319.144560013 podStartE2EDuration="5m24.42889413s" totalImagesPullingTime="5.284334117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:22.311591042 +0000 UTC m=+37.871094012" watchObservedRunningTime="2026-10-31 03:19:22.42889413 +0000 UTC m=+37.988397101" Oct 31 03:19:22.429153 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.429028 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pvphc" podStartSLOduration=2.868222157 podStartE2EDuration="8.42902037s" totalImagesPullingTime="5.560798213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:22.428774 +0000 UTC m=+37.988276972" watchObservedRunningTime="2026-10-31 03:19:22.42902037 +0000 UTC m=+37.988523340" Oct 31 03:19:22.547332 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.547277 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" podStartSLOduration=320.081639613 podStartE2EDuration="5m25.547261093s" totalImagesPullingTime="5.46562148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:22.545744675 +0000 UTC m=+38.105247646" watchObservedRunningTime="2026-10-31 03:19:22.547261093 +0000 UTC m=+38.106764071" Oct 31 03:19:22.630504 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.630392 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z7s46"] Oct 31 03:19:22.650624 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.650549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.667073 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.666537 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zrtsh" podStartSLOduration=298.355218659 podStartE2EDuration="5m3.666520246s" totalImagesPullingTime="5.311301587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:22.665664254 +0000 UTC m=+38.225167226" watchObservedRunningTime="2026-10-31 03:19:22.666520246 +0000 UTC m=+38.226023217" Oct 31 03:19:22.687778 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.687740 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z7s46"] Oct 31 03:19:22.691707 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.691342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c58fbd8-7809-4718-8137-3439ac40c593-data-volume\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.691707 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.691437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c58fbd8-7809-4718-8137-3439ac40c593-crio-socket\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.691707 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.691479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.691707 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.691521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nt2\" (UniqueName: \"kubernetes.io/projected/6c58fbd8-7809-4718-8137-3439ac40c593-kube-api-access-j8nt2\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.691707 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.691554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.704028 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.704004 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:19:22.704157 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.704113 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:19:22.704207 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.704151 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dbgqm\"" Oct 31 03:19:22.792874 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.792833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:22.793045 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.792893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:22.793045 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.792925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:22.793045 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.792955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:22.793045 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.792986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:22.793263 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793156 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:22.793471 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793188 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:22.793471 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793434 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:22.793471 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793187 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793203 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.793181456 +0000 UTC m=+46.352684407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793514 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.793495082 +0000 UTC m=+46.352998032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793532 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.793522199 +0000 UTC m=+46.353025146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793545 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.793537307 +0000 UTC m=+46.353040254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793263 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793576 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.793568983 +0000 UTC m=+46.353071968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c58fbd8-7809-4718-8137-3439ac40c593-data-volume\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.793656 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c58fbd8-7809-4718-8137-3439ac40c593-data-volume\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c58fbd8-7809-4718-8137-3439ac40c593-crio-socket\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c58fbd8-7809-4718-8137-3439ac40c593-crio-socket\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793861 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793889 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793908 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:23.293898327 +0000 UTC m=+38.853401287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793949 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.793933083 +0000 UTC m=+46.353436047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.793984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nt2\" (UniqueName: \"kubernetes.io/projected/6c58fbd8-7809-4718-8137-3439ac40c593-kube-api-access-j8nt2\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.793994 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.794002 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.794020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.794037 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.794025 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.794017418 +0000 UTC m=+46.353520397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:22.794740 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.794134 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:22.794740 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.794171 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:23.294158176 +0000 UTC m=+38.853661138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : configmap "kube-rbac-proxy" not found Oct 31 03:19:22.856675 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.856647 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nt2\" (UniqueName: \"kubernetes.io/projected/6c58fbd8-7809-4718-8137-3439ac40c593-kube-api-access-j8nt2\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:22.895643 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:22.895423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:22.895643 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.895556 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:22.895643 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:22.895618 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.895599566 +0000 UTC m=+46.455102519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:23.082279 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.082221 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr"] Oct 31 03:19:23.102129 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.102093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" Oct 31 03:19:23.143517 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.143482 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr"] Oct 31 03:19:23.158569 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.158533 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 03:19:23.158757 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.158691 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b5nvx\"" Oct 31 03:19:23.158831 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.158784 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:23.194540 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.194476 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" event={"ID":"2901d41f-f26c-4e6b-909f-1fc449e11ce8","Type":"ContainerStarted","Data":"490b489e0ea40f98d7e5d19af83a752e73752da54af52f2ebea1b6cbfbde0123"} Oct 31 03:19:23.196511 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.196492 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/1.log" Oct 31 03:19:23.196992 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.196976 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/0.log" Oct 31 03:19:23.197103 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.197012 2575 generic.go:356] "Generic (PLEG): container finished" podID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" containerID="94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4" exitCode=255 Oct 31 03:19:23.197103 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.197082 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" event={"ID":"9bdf0731-bef7-4c9c-a5c3-09eff5aef394","Type":"ContainerDied","Data":"94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4"} Oct 31 03:19:23.197584 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.197158 2575 scope.go:122] "RemoveContainer" containerID="b0df75062e850088f77634f43a4abb72d7f3c30b82a8fa881e6cb58031454bb1" Oct 31 03:19:23.197584 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.197393 2575 scope.go:122] "RemoveContainer" containerID="94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4" Oct 31 03:19:23.197754 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:23.197588 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:23.198336 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.198317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4tm6\" (UniqueName: \"kubernetes.io/projected/d0e050b4-e196-498c-bb05-3b1bc20f10ae-kube-api-access-l4tm6\") pod \"migrator-66df5d496b-nv8wr\" (UID: \"d0e050b4-e196-498c-bb05-3b1bc20f10ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" Oct 31 03:19:23.299875 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.299214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4tm6\" (UniqueName: \"kubernetes.io/projected/d0e050b4-e196-498c-bb05-3b1bc20f10ae-kube-api-access-l4tm6\") pod \"migrator-66df5d496b-nv8wr\" (UID: \"d0e050b4-e196-498c-bb05-3b1bc20f10ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" Oct 31 03:19:23.299875 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.299424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:23.299875 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.299523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:23.299875 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:23.299690 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:23.299875 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:23.299750 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:24.299730815 +0000 UTC m=+39.859233780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:23.299875 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:23.299792 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:23.299875 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:23.299828 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:24.299816645 +0000 UTC m=+39.859319611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : configmap "kube-rbac-proxy" not found Oct 31 03:19:23.361857 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.361789 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4tm6\" (UniqueName: \"kubernetes.io/projected/d0e050b4-e196-498c-bb05-3b1bc20f10ae-kube-api-access-l4tm6\") pod \"migrator-66df5d496b-nv8wr\" (UID: \"d0e050b4-e196-498c-bb05-3b1bc20f10ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" Oct 31 03:19:23.428854 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.428813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" Oct 31 03:19:23.468888 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.468840 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" podStartSLOduration=298.12611477 podStartE2EDuration="5m4.468825891s" totalImagesPullingTime="6.342711121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:23.467200345 +0000 UTC m=+39.026703337" watchObservedRunningTime="2026-10-31 03:19:23.468825891 +0000 UTC m=+39.028328861" Oct 31 03:19:23.824132 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:23.824107 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr"] Oct 31 03:19:23.826272 ip-10-0-138-238 kubenswrapper[2575]: W1031 03:19:23.826242 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e050b4_e196_498c_bb05_3b1bc20f10ae.slice/crio-53ec70cb09a8678c05ca51e58b777087c00111deebef8f899c008bc2d7fdb68e WatchSource:0}: Error finding container 53ec70cb09a8678c05ca51e58b777087c00111deebef8f899c008bc2d7fdb68e: Status 404 returned error can't find the container with id 53ec70cb09a8678c05ca51e58b777087c00111deebef8f899c008bc2d7fdb68e Oct 31 03:19:24.202282 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.202186 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/1.log" Oct 31 03:19:24.202712 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.202587 2575 scope.go:122] "RemoveContainer" containerID="94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4" Oct 31 03:19:24.202814 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:24.202789 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:24.203743 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.203706 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" event={"ID":"d0e050b4-e196-498c-bb05-3b1bc20f10ae","Type":"ContainerStarted","Data":"53ec70cb09a8678c05ca51e58b777087c00111deebef8f899c008bc2d7fdb68e"} Oct 31 03:19:24.205429 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.205396 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wdw6h" event={"ID":"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d","Type":"ContainerStarted","Data":"5036208c6f7ccc2eb6863d48065b6656571572b03f6b4844c6e57b0808513401"} Oct 31 03:19:24.205742 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.205719 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:24.309327 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.309296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:24.309514 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.309365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:24.309514 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:24.309468 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:24.309627 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:24.309539 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:26.309518818 +0000 UTC m=+41.869021782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:24.309913 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:24.309888 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:24.310046 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:24.309933 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:26.30992143 +0000 UTC m=+41.869424408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : configmap "kube-rbac-proxy" not found Oct 31 03:19:24.438969 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:24.438904 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wdw6h" podStartSLOduration=35.678529864 podStartE2EDuration="39.438885923s" totalImagesPullingTime="3.760356059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:24.43776105 +0000 UTC m=+39.997264022" watchObservedRunningTime="2026-10-31 03:19:24.438885923 +0000 UTC m=+39.998388927" Oct 31 03:19:25.333134 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:25.333084 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:25.333639 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:25.333163 2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:25.333639 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:25.333589 2575 scope.go:122] "RemoveContainer" containerID="94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4" Oct 31 03:19:25.333811 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:25.333782 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:26.213278 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:26.213173 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" event={"ID":"d0e050b4-e196-498c-bb05-3b1bc20f10ae","Type":"ContainerStarted","Data":"6153bdb056cb248f3c597d91e778c1382c906a66c1e8e6e948a10a62a20b9319"} Oct 31 03:19:26.330654 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:26.330606 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" podStartSLOduration=1.230340114 podStartE2EDuration="3.330592147s" totalImagesPullingTime="2.100252033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:26.329318445 +0000 UTC m=+41.888821415" watchObservedRunningTime="2026-10-31 03:19:26.330592147 +0000 UTC m=+41.890095116" Oct 31 03:19:26.331312 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:26.331282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:26.331434 ip-10-0-138-238 kubenswrapper[2575]: I1031 03:19:26.331355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:26.331511 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:26.331490 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:26.331566 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:26.331537 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:26.331566 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:26.331557 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.331541189 +0000 UTC m=+45.891044137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:26.331648 ip-10-0-138-238 kubenswrapper[2575]: E1031 03:19:26.331587 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.331573063 +0000 UTC m=+45.891076029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : configmap "kube-rbac-proxy" not found Oct 31 03:19:26.621805 ip-10-0-138-238 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 03:19:26.637650 ip-10-0-138-238 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 03:19:26.637877 ip-10-0-138-238 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 03:19:26.638108 ip-10-0-138-238 systemd[1]: kubelet.service: Consumed 3.208s CPU time. Oct 31 03:19:26.649701 ip-10-0-138-238 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:19:26.740142 ip-10-0-138-238 kubenswrapper[4934]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:19:26.740142 ip-10-0-138-238 kubenswrapper[4934]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:19:26.740142 ip-10-0-138-238 kubenswrapper[4934]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:19:26.740142 ip-10-0-138-238 kubenswrapper[4934]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:19:26.742452 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742370 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.742452 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742452 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.742452 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742456 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742459 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742462 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742465 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742468 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742471 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742474 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742476 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742479 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742481 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742483 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742486 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742488 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742492 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742494 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742497 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742499 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742502 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742504 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742511 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.742548 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742514 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742516 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742523 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742527 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742530 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742533 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742536 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742539 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742541 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742544 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742546 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742549 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742551 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742554 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742557 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742560 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742562 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742566 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742568 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.743028 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742571 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742573 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742576 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742578 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742581 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742583 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742585 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742588 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742590 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742593 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742595 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742598 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742600 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742603 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742605 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742609 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742611 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742614 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742617 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742620 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.743521 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742622 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742625 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742627 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742630 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742632 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742634 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742637 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742639 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742642 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742644 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742647 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742650 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742653 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742656 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742658 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742661 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742663 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742666 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742668 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742671 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.744059 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742673 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742676 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742679 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742681 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742685 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742690 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742692 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742697 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742699 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742702 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742704 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742707 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742709 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742711 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742714 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742717 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742719 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742722 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742724 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.744582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742727 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742729 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742731 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742734 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742736 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.742739 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743553 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743561 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743565 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743567 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743570 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743572 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743575 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743578 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743581 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743583 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743586 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743589 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743591 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743594 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.745041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743597 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743599 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743602 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743604 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743607 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743610 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743612 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743614 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743617 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743619 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743622 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743624 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743627 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743629 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743632 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743634 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743637 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743639 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743642 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743645 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.745536 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743648 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743650 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743653 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743655 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743657 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743660 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743663 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743665 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743668 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743670 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743673 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743675 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743677 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743680 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743683 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743685 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743688 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743690 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743693 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.746023 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743695 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743698 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743700 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743702 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743705 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743708 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743710 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743712 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743715 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743717 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743719 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743722 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743731 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743733 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743736 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743738 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743741 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743743 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743747 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743750 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.746529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743752 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743755 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743758 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743760 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743763 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743765 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743768 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743770 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743773 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743775 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743778 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743780 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743783 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743785 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743787 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743790 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743792 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743795 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743797 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743800 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.747041 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743802 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743807 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743810 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743813 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743817 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743822 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743825 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743828 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743832 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743836 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743839 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743842 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.743844 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744382 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744388 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744392 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744394 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744397 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744400 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.747537 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744402 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744405 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744407 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744410 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744412 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744415 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744418 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744420 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744423 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744425 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744428 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744430 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744433 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744435 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744440 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744443 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744445 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744448 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744451 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.748008 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744453 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744457 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744459 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744462 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744464 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744467 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744469 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744472 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744474 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744478 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744482 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744484 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744487 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744490 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744492 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744495 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744497 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744500 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744502 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.748529 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744505 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744507 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744510 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744512 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744515 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744517 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744520 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744522 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744526 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744529 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744531 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744534 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744536 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744539 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744542 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744544 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744547 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744549 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744552 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.748990 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744555 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744557 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744559 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744562 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744564 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744567 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744572 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744575 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744578 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744581 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744584 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744587 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744590 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744593 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744595 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744598 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744602 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744605 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744607 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.749458 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744609 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744612 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744616 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744619 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744621 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744623 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744626 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744629 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744632 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744634 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744637 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744639 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744642 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744644 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744647 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744649 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744652 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744654 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744658 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744661 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.749945 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744663 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744665 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744668 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.744670 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744768 4934 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744784 4934 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744794 4934 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744801 4934 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744807 4934 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744812 4934 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744816 4934 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744820 4934 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744823 4934 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744826 4934 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744831 4934 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744834 4934 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744837 4934 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744840 4934 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744843 4934 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744846 4934 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744849 4934 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744852 4934 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744856 4934 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744859 4934 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:19:26.750485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744862 4934 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744865 4934 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744869 4934 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744873 4934 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744876 4934 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744879 4934 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744882 4934 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744885 4934 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744888 4934 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744891 4934 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744894 4934 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744897 4934 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744901 4934 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744904 4934 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744907 4934 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744910 4934 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744913 4934 flags.go:64] FLAG: --enable-server="true" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744917 4934 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744922 4934 flags.go:64] FLAG: --event-burst="100" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744925 4934 flags.go:64] FLAG: --event-qps="50" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744928 4934 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744931 4934 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744935 4934 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744940 4934 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:19:26.751064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744943 4934 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744946 4934 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744949 4934 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744952 4934 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744956 4934 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744959 4934 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744962 4934 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744965 4934 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744968 4934 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744971 4934 flags.go:64] FLAG: --feature-gates="" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744975 4934 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744979 4934 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744982 4934 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744985 4934 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744988 4934 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744991 4934 flags.go:64] FLAG: --help="false" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744994 4934 flags.go:64] FLAG: --hostname-override="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.744997 4934 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745000 4934 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745003 4934 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745007 4934 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745010 4934 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745013 4934 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:19:26.752139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745016 4934 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745019 4934 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745023 4934 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745026 4934 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745029 4934 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745032 4934 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745035 4934 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745039 4934 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745042 4934 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745044 4934 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745047 4934 flags.go:64] FLAG: --lock-file="" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745050 4934 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745053 4934 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745057 4934 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745062 4934 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745065 4934 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745069 4934 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745071 4934 flags.go:64] FLAG: --logging-format="text" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745074 4934 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745078 4934 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745080 4934 flags.go:64] FLAG: --manifest-url="" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745083 4934 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745088 4934 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745091 4934 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745094 4934 flags.go:64] FLAG: --max-pods="110" Oct 31 03:19:26.752849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745097 4934 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745100 4934 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745103 4934 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745106 4934 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745109 4934 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745112 4934 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745115 4934 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745121 4934 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745124 4934 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745129 4934 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745132 4934 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745135 4934 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745138 4934 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745141 4934 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745145 4934 flags.go:64] FLAG: --port="10250" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745148 4934 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745151 4934 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01d83ca4bf958a366" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745154 4934 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745157 4934 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745160 4934 flags.go:64] FLAG: --register-node="true" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745163 4934 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745167 4934 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745170 4934 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745173 4934 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745176 4934 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745180 4934 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:19:26.753468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745183 4934 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745186 4934 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745189 4934 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745192 4934 flags.go:64] FLAG: --runonce="false" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745195 4934 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745197 4934 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745200 4934 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745203 4934 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745206 4934 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745209 4934 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745212 4934 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745215 4934 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745218 4934 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745221 4934 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745237 4934 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745241 4934 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745244 4934 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745247 4934 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745250 4934 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745255 4934 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745259 4934 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745262 4934 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745266 4934 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745269 4934 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745271 4934 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:19:26.754114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745275 4934 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745278 4934 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745281 4934 flags.go:64] FLAG: --v="2" Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745285 4934 flags.go:64] FLAG: --version="false" Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745289 4934 flags.go:64] FLAG: --vmodule="" Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745293 4934 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745297 4934 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745457 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745461 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745466 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745470 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745473 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745475 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745478 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745481 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745483 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745486 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745488 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745490 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745499 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745502 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.754726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745504 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745508 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745511 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745513 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745516 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745519 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745523 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745526 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745528 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745531 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745533 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745536 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745540 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745545 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745548 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745551 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745554 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745556 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745559 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745562 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.755567 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745564 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745567 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745569 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745572 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745574 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745577 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745579 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745582 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745584 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745587 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745589 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745592 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745594 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745598 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745600 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745603 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745605 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745608 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745611 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745613 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.756066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745616 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745618 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745620 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745623 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745625 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745628 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745630 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745632 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745635 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745637 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745640 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745643 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745645 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745648 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745650 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745653 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745655 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745657 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745660 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.756729 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745663 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745665 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745667 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745670 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745672 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745675 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745678 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745681 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745683 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745685 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745688 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745691 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745693 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745695 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745698 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745700 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745714 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745718 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745721 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745724 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.757341 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745726 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745729 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745731 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745734 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745736 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745739 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745742 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745744 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745747 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745749 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745751 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745754 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745756 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.745784 4934 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745827 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745831 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.758217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745834 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745836 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745841 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745843 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745846 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745849 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745851 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745854 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745856 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745859 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745861 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745865 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745869 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745872 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745875 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745877 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745880 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745882 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745885 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.758895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745887 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745889 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745892 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745895 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745897 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745900 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745902 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745905 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745907 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745910 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745912 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745914 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745917 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745920 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745922 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745926 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745928 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745931 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745933 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745936 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.759511 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745938 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745940 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745943 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745945 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745948 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745950 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745952 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745955 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745957 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745960 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745962 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745965 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745967 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745970 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745972 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745974 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745978 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745981 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745984 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.760043 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745986 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745989 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745993 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745996 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.745998 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746000 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746003 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746005 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746009 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746011 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746014 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746017 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746019 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746022 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746024 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746027 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746030 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746032 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746035 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746038 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.760543 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746040 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746042 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746045 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746047 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746057 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746061 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746063 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746066 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746068 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746071 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746074 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746077 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746079 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746082 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746085 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746090 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746094 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746096 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746099 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.761021 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746101 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746105 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746107 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746110 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746113 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746115 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.746117 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.750310 4934 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.750326 4934 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.750342 4934 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.750347 4934 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750417 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750423 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750425 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750429 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750432 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750434 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750437 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750440 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750443 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.761483 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750446 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750448 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750451 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750460 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750463 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750465 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750468 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750471 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750474 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750477 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750480 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750482 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750485 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750488 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750490 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750493 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750496 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750498 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750502 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750504 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.761946 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750507 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750509 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750512 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750514 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750517 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750519 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750522 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750524 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750527 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750529 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750531 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750534 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750537 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750539 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750542 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750544 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750548 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750550 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750553 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750556 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.762454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750559 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750561 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750563 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750566 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750568 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750571 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750573 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750576 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750578 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750581 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750583 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750586 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750588 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750590 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750593 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750596 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750599 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750601 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750604 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.762929 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750606 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750610 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750614 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750617 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750620 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750624 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750627 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750631 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750634 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750637 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750643 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750646 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750648 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750651 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750654 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750656 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750659 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750661 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750664 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.763448 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750666 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750669 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750671 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750673 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750676 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750678 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750681 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750683 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750687 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750689 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750692 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750694 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750697 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750699 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750702 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750705 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750707 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750710 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.763942 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750712 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.750739 4934 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750957 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750963 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750966 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.750969 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751010 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751135 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751142 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751146 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751151 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751156 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751160 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751165 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751169 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751174 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.764384 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751179 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751183 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751188 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751193 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751202 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751207 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751211 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751216 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751220 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751248 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751253 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751257 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751262 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751267 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751272 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751276 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751281 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751290 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751295 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.764757 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751299 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751303 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751308 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751313 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751318 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751323 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751327 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751331 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751335 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751340 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751349 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751353 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751357 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751362 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751366 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751370 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751374 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751381 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751386 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751391 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.765207 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751396 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751400 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751405 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751414 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751419 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751424 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751430 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751437 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751442 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751447 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751452 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751456 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751460 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751465 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751469 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751478 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751482 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751487 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751491 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.765694 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751496 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751500 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751504 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751508 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751512 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751516 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751520 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751524 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751528 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751537 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751541 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751545 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751549 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751554 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751558 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751562 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751566 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751571 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751575 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.766217 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751579 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751584 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751588 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751597 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751601 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751605 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751609 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751613 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751617 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751621 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751625 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751629 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751635 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751639 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751644 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.751694 4934 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:19:26.766690 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751825 4934 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751833 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751838 4934 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751847 4934 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751851 4934 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751855 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751860 4934 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751870 4934 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751875 4934 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751879 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751883 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751888 4934 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751892 4934 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751896 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751902 4934 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751914 4934 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751919 4934 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751924 4934 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751928 4934 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751932 4934 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:26.767069 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751936 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751941 4934 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751945 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751952 4934 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751957 4934 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751962 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751967 4934 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751976 4934 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751980 4934 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751985 4934 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751990 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751994 4934 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.751998 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752002 4934 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752006 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752010 4934 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752015 4934 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752019 4934 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752023 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:26.767591 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752029 4934 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752038 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752042 4934 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752046 4934 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752050 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752054 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752058 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752062 4934 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752066 4934 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752070 4934 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752074 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752079 4934 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752083 4934 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752093 4934 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752097 4934 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752102 4934 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752106 4934 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752109 4934 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752114 4934 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:26.768191 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752118 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752122 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752126 4934 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752130 4934 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752135 4934 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752139 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752144 4934 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752154 4934 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752159 4934 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752163 4934 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752167 4934 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752172 4934 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752176 4934 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752182 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752186 4934 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752190 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752194 4934 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752199 4934 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752203 4934 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752293 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:26.768948 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752311 4934 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752317 4934 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752322 4934 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752327 4934 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752332 4934 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752337 4934 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752341 4934 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752346 4934 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752350 4934 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752355 4934 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752359 4934 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752364 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752373 4934 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752378 4934 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752382 4934 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752386 4934 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752391 4934 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752396 4934 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752400 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:26.769555 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752408 4934 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752413 4934 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752433 4934 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752438 4934 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752443 4934 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752551 4934 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752563 4934 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752566 4934 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:26.752569 4934 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.752657 4934 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.753666 4934 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.753720 4934 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.754011 4934 server.go:1014] "Starting client certificate rotation" Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.754171 4934 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.754256 4934 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:13:45 +0000 UTC" deadline="2028-06-11 03:19:46.652723117 +0000 UTC" Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.754293 4934 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14136h0m19.898433072s" Oct 31 03:19:26.770004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.754590 4934 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:19:26.770428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.755708 4934 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:19:26.770428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.756885 4934 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:19:26.770428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.758969 4934 log.go:25] "Validated CRI v1 image API" Oct 31 03:19:26.770428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.759841 4934 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:19:26.770428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.761753 4934 fs.go:135] Filesystem UUIDs: map[5b85a941-1bf5-48de-8b4c-0034d8475172:/dev/nvme0n1p3 6aea7901-7c47-426c-b91a-7ea692c015c5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 31 03:19:26.770571 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.761768 4934 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19a9a3b840f5998529e86e283fa2ea71831b50665e81d68008d91f0b4897fca9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19a9a3b840f5998529e86e283fa2ea71831b50665e81d68008d91f0b4897fca9/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38b5a4b1d9e003591d385e70a51f3306a15cae63ce0f9be186bced90abe010f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38b5a4b1d9e003591d385e70a51f3306a15cae63ce0f9be186bced90abe010f3/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b1ba02d33e619ce2f9882279d307b46171da0e12c021ffce8557a14b78916a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b1ba02d33e619ce2f9882279d307b46171da0e12c021ffce8557a14b78916a6/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/437e8d737b8b666cb4aceb772a4327c2d99f84bbe2721c1799c57d7e1ce90a81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/437e8d737b8b666cb4aceb772a4327c2d99f84bbe2721c1799c57d7e1ce90a81/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c3ac7e10b8a12eea0997eefb37a76c9fdd4453eb1a87cc0077a9d620241c91e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c3ac7e10b8a12eea0997eefb37a76c9fdd4453eb1a87cc0077a9d620241c91e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52b0684eb07236bb6cbe33e42f936e5aead498ee528359dc2d37bf332b6d746d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52b0684eb07236bb6cbe33e42f936e5aead498ee528359dc2d37bf332b6d746d/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53e025546c07d72f5d9533e39a5f13c0fba1524ee57f8f3f0d8de768ae7b67f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53e025546c07d72f5d9533e39a5f13c0fba1524ee57f8f3f0d8de768ae7b67f2/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53ec70cb09a8678c05ca51e58b777087c00111deebef8f899c008bc2d7fdb68e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53ec70cb09a8678c05ca51e58b777087c00111deebef8f899c008bc2d7fdb68e/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77907b880c28799f8c97c0150d5ca25dbbbe3c1534e2e5cbb8eaed6ed712c17c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77907b880c28799f8c97c0150d5ca25dbbbe3c1534e2e5cbb8eaed6ed712c17c/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7befa522b24d9bfa6b0232bb97c32b94861ad61222826e8fa39ad6abc0dd4807/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7befa522b24d9bfa6b0232bb97c32b94861ad61222826e8fa39ad6abc0dd4807/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c3b90117d898e769619cdec74f18d1e69f25206f4b6d63d100b8c6e5dd29e59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c3b90117d898e769619cdec74f18d1e69f25206f4b6d63d100b8c6e5dd29e59/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86348cd6e950a49f812da1f9212057a5cc992ab5d5cbc68b6bf780a54031b307/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86348cd6e950a49f812da1f9212057a5cc992ab5d5cbc68b6bf780a54031b307/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bffa4ab83c06e166284c2b70616242842d5edae65e73d11961aefbfdd1cf7c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bffa4ab83c06e166284c2b70616242842d5edae65e73d11961aefbfdd1cf7c7/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1e1ea2182e46ce3b256ed3cc00bb75a1345a7cab033c2ce5cb8689a76cc3b10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1e1ea2182e46ce3b256ed3cc00bb75a1345a7cab033c2ce5cb8689a76cc3b10/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c22d8d4598865ed32963a35212cc2a10673b9d9fbb9984199c6adf4cd6cdaaf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c22d8d4598865ed32963a35212cc2a10673b9d9fbb9984199c6adf4cd6cdaaf3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2e6ccf3cf471d62cd75c792e3ef1f94f53c4af0f747896364dcf8a8d962f6e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2e6ccf3cf471d62cd75c792e3ef1f94f53c4af0f747896364dcf8a8d962f6e2/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dba80bdeb3e79704bc274388bf14fccf7d5b4981c72c4d0f2c95b5505c96e3a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dba80bdeb3e79704bc274388bf14fccf7d5b4981c72c4d0f2c95b5505c96e3a3/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e293a71db3bd5f844f1fe493ed4af34894e79aaf3abd77b346b8fc75dedbd983/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e293a71db3bd5f844f1fe493ed4af34894e79aaf3abd77b346b8fc75dedbd983/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5419e24989d40d3a07f438241f3aab44261dc54a0f4a379a2a1c2e63daa8fa1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5419e24989d40d3a07f438241f3aab44261dc54a0f4a379a2a1c2e63daa8fa1/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcd73e7ab4ad36809a42fd963afd3fbd9d2a4ccd86d02336ca45de5495430799/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcd73e7ab4ad36809a42fd963afd3fbd9d2a4ccd86d02336ca45de5495430799/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe6c8824f78c4b17885dfe6daaf9774b4f1bc7a68260cb6bfed3971a24e588dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe6c8824f78c4b17885dfe6daaf9774b4f1bc7a68260cb6bfed3971a24e588dc/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15c8c5bb-10f1-4894-b45d-34b2209f9676/volumes/kubernetes.io~projected/kube-api-access-nnfr7:{mountpoint:/var/lib/kubelet/pods/15c8c5bb-10f1-4894-b45d-34b2209f9676/volumes/kubernetes.io~projected/kube-api-access-nnfr7 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1687ced2-8f63-4900-9b49-275a8104e159/volumes/kubernetes.io~projected/kube-api-access-rsdh2:{mountpoint:/var/lib/kubelet/pods/1687ced2-8f63-4900-9b49-275a8104e159/volumes/kubernetes.io~projected/kube-api-access-rsdh2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c5fbb4c-6297-4655-8c67-ff336a28e948/volumes/kubernetes.io~projected/kube-api-access-74zsl:{mountpoint:/var/lib/kubelet/pods/1c5fbb4c-6297-4655-8c67-ff336a28e948/volumes/kubernetes.io~projected/kube-api-access-74zsl major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/283b8014-5ffd-4671-a28d-2975376d9cf9/volumes/kubernetes.io~projected/kube-api-access-kkjmw:{mountpoint:/var/lib/kubelet/pods/283b8014-5ffd-4671-a28d-2975376d9cf9/volumes/kubernetes.io~projected/kube-api-access-kkjmw major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2901d41f-f26c-4e6b-909f-1fc449e11ce8/volumes/kubernetes.io~projected/kube-api-access-s8xtp:{mountpoint:/var/lib/kubelet/pods/2901d41f-f26c-4e6b-909f-1fc449e11ce8/volumes/kubernetes.io~projected/kube-api-access-s8xtp major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90/volumes/kubernetes.io~projected/kube-api-access-5xj84:{mountpoint:/var/lib/kubelet/pods/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90/volumes/kubernetes.io~projected/kube-api-access-5xj84 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a5ed6d1-a843-4d21-b913-59b0f9221e42/volumes/kubernetes.io~projected/kube-api-access-8ww68:{mountpoint:/var/lib/kubelet/pods/3a5ed6d1-a843-4d21-b913-59b0f9221e42/volumes/kubernetes.io~projected/kube-api-access-8ww68 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3aed8bd2-021f-4e33-96f0-e9c974fa0701/volumes/kubernetes.io~projected/kube-api-access-2n2t5:{mountpoint:/var/lib/kubelet/pods/3aed8bd2-021f-4e33-96f0-e9c974fa0701/volumes/kubernetes.io~projected/kube-api-access-2n2t5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3aed8bd2-021f-4e33-96f0-e9c974fa0701/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3aed8bd2-021f-4e33-96f0-e9c974fa0701/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cfceaec-d546-4c9e-ac02-e66eeeea477a/volumes/kubernetes.io~projected/kube-api-access-qlwvt:{mountpoint:/var/lib/kubelet/pods/3cfceaec-d546-4c9e-ac02-e66eeeea477a/volumes/kubernetes.io~projected/kube-api-access-qlwvt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eefb160-a377-431f-be0a-c2bea6ecb352/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3eefb160-a377-431f-be0a-c2bea6ecb352/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6/volumes/kubernetes.io~projected/kube-api-access-w8jgl:{mountpoint:/var/lib/kubelet/pods/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6/volumes/kubernetes.io~projected/kube-api-access-w8jgl major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~projected/kube-api-access-n27js:{mountpoint:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~projected/kube-api-access-n27js major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69dbc1e2-7632-40e1-8e81-79cf08b42d67/volumes/kubernetes.io~projected/kube-api-access-jxtgd:{mountpoint:/var/lib/kubelet/pods/69dbc1e2-7632-40e1-8e81-79cf08b42d67/volumes/kubernetes.io~projected/kube-api-access-jxtgd major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c58fbd8-7809-4718-8137-3439ac40c593/volumes/kubernetes.io~projected/kube-api-access-j8nt2:{mountpoint:/var/lib/kubelet/pods/6c58fbd8-7809-4718-8137-3439ac40c593/volumes/kubernetes.io~projected/kube-api-access-j8nt2 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~projected/kube-api-access-jmczk:{mountpoint:/var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~projected/kube-api-access-jmczk major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93cfa49d-0426-4d23-ad84-50f12272a871/volumes/kubernetes.io~projected/kube-api-access-sppr9:{mountpoint:/var/lib/kubelet/pods/93cfa49d-0426-4d23-ad84-50f12272a871/volumes/kubernetes.io~projected/kube-api-access-sppr9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bdf0731-bef7-4c9c-a5c3-09eff5aef394/volumes/kubernetes.io~projected/kube-api-access-xvlc5:{mountpoint:/var/lib/kubelet/pods/9bdf0731-bef7-4c9c-a5c3-09eff5aef394/volumes/kubernetes.io~projected/kube-api-access-xvlc5 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bdf0731-bef7-4c9c-a5c3-09eff5aef394/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9bdf0731-bef7-4c9c-a5c3-09eff5aef394/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ec0a926-6081-4a94-916a-2e27400b06a9/volumes/kubernetes.io~projected/kube-api-access-trx4z:{mountpoint:/var/lib/kubelet/pods/9ec0a926-6081-4a94-916a-2e27400b06a9/volumes/kubernetes.io~projected/kube-api-access-trx4z major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f6df5b8-3430-46ee-8af1-13bb15bda816/volumes/kubernetes.io~projected/kube-api-access-xs2ps:{mountpoint:/var/lib/kubelet/pods/9f6df5b8-3430-46ee-8af1-13bb15bda816/volumes/kubernetes.io~projected/kube-api-access-xs2ps major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f6df5b8-3430-46ee-8af1-13bb15bda816/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9f6df5b8-3430-46ee-8af1-13bb15bda816/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a57d414f-4fb4-4b78-92c7-105e2ac19d21/volumes/kubernetes.io~projected/kube-api-access-xfsgb:{mountpoint:/var/lib/kubelet/pods/a57d414f-4fb4-4b78-92c7-105e2ac19d21/volumes/kubernetes.io~projected/kube-api-access-xfsgb major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a57d414f-4fb4-4b78-92c7-105e2ac19d21/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a57d414f-4fb4-4b78-92c7-105e2ac19d21/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad09c972-be06-48b1-84b0-a25fc1c645ba/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ad09c972-be06-48b1-84b0-a25fc1c645ba/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~projected/kube-api-access-xzpcf:{mountpoint:/var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~projected/kube-api-access-xzpcf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6c13c0-234d-4042-a31c-ac2c374fd1b3/volumes/kubernetes.io~projected/kube-api-access-7mqds:{mountpoint:/var/lib/kubelet/pods/ba6c13c0-234d-4042-a31c-ac2c374fd1b3/volumes/kubernetes.io~projected/kube-api-access-7mqds major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdd9600b-d5c9-4add-ae37-48aca3854510/volumes/kubernetes.io~projected/kube-api-access-z7scj:{mountpoint:/var/lib/kubelet/pods/bdd9600b-d5c9-4add-ae37-48aca3854510/volumes/kubernetes.io~projected/kube-api-access-z7scj major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~projected/kube-api-access-58zth:{mountpoint:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~projected/kube-api-access-58zth major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6c807d9-708e-4527-8e56-11749a26ca61/volumes/kubernetes.io~projected/kube-api-access-phhzp:{mountpoint:/var/lib/kubelet/pods/c6c807d9-708e-4527-8e56-11749a26ca61/volumes/kubernetes.io~projected/kube-api-access-phhzp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2/volumes/kubernetes.io~projected/kube-api-access-zqpm8:{mountpoint:/var/lib/kubelet/pods/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2/volumes/kubernetes.io~projected/kube-api-access-zqpm8 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volumes/kubernetes.io~projected/kube-api-access-vzp8s:{mountpoint:/var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volumes/kubernetes.io~projected/kube-api-access-vzp8s major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e050b4-e196-498c-bb05-3b1bc20f10ae/volumes/kubernetes.io~projected/kube-api-access-l4tm6:{mountpoint:/var/lib/kubelet/pods/d0e050b4-e196-498c-bb05-3b1bc20f10ae/volumes/kubernetes.io~projected/kube-api-access-l4tm6 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d/volumes/kubernetes.io~projected/kube-api-access-25l6m:{mountpoint:/var/lib/kubelet/pods/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d/volumes/kubernetes.io~projected/kube-api-access-25l6m major:0 minor:288 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/75e7a4fa3841e9e32619f83189849987fea872836f348fbc83ac31c52586027b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/814663296c9eaec1c2840960bad6fbc6397443d40d19a6dc588ad6d710b8581a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c3a919c306b9f56c0d968873349984c94afc9425c13cdcec7e1d6e5679ee863d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/f48f24173ac0b9c35847963caf35f96f393a33d0fc1a90b83603a9f85278347c/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e9e0a6b17769d14d3b29478a671dba8db80b89a8a086db064ebd1fcc685dcb7f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5104b10993eff82d840bd41d2bad03cf9317f7235af743087081015bf6171fb0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ae0cebbba9f7f9cbf50731576304683be45e9ab3d830cf64a925545a3f9c97ea/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d89aa2b8f22953940dc66a3123e17c87f62ea99e9c0bceb99308b7091ce37e45/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/d2883f7485884d085ddc19129dab4f025d47925d397cee9f9929699590bd0897/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/a87e78c464f6c4d0e3b03e8ebbde35bb09cc784c793ddf40670d4836f9ec88ba/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2ffa5b9b9a23ef3cd1371a64de47e0df57775f0ba8a0ca59aead0b689d87e3d5/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3e7ec9acf38bd596479c564fabfdfc332ff25791b41a7655495b526527408988/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/97b8a0fc075bc591e87d2c11816a5e3cdc1fd09a4d5e7618bc603d86d079e34b/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a5d2bee834b324ef532100408b29bfcfcac74311c79c465908374b4e97fc5d40/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9440a0d0893a1c67b2cb65ba5c121cdef746e857ac86614cb28aadb089eb7b4c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f22402fbcc14614e85c634933582a28bc792fadb2082ef44454cd9d81861a703/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8445175c085006c90035b4be8ad056b4a55b02b089ed14165e387eab02e90647/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2eb684109ba6c1f2b29b0bb3e7a1c85539d04e2ba740f502ec4a5eb9e08386d5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2b0a0fc061553feb4f6995cdbe0baa7cf3e0898b8a019cd5eb0f2e9e9e8dc358/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9a325a80501ccf6912a6d495bc88cde8b92f1c1100726e8a0d26635f8d61f66c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/beebd636795484867d41e4c775908bd72d178dfada3310ea877dc49e14f52252/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/c2baecb17b8e2a636217a61036bc3a077554404f933f6a63a4bc64e129395b3d/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/73bafeaf99cebc4b13ffd268c8ba931b3ccd31246b2680e2eca9d59b7f2878f4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/9fb426cd0790d04cb0cc1b9d2f47f2b06d78c8a3385157db612dd59d975d256c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f57bb2c2d745e10267d85f4d6891002cfe02e67b9da533a64a27cbdb56b16f01/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/dc7f67d9c92f73cfed4cce8b51aa0a261da8a63b2f404adabe48aa8ef08416bf/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/fa6c5d609ef3f379c6870e435ff87ce0dea10a224875774b8e873b28de1996ec/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/f9cc2c5e0e064f3e43507fbb444b83f40a02bea413ba0f6cf0174c32e9466ab3/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b61af8a000bd7705d7d321edd89696ce29af0141d66c8c502240816ea14fbdc8/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/51a3c01a57ccf444f2b158690825ab89f58fc135a37194490061b04c85354eee/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/ceb7375fd63cc81100ac51a6da3dedf5fc33329a6fcabae37b0ba689e41d182a/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/3c903feab9cad56ab90c2f5b66d5a074f988088c9bed5f9c0c1528d1be41a3db/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/500cd9f30cc1e13a47cdaba988e5c7c5dcaa288821404c1e1d65ad3ef8d51c28/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/ae27f295b1bd5166b303229115b5804b29d9569be6af372b072f2f775725dd0e/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/e0098a7810bc804d2d10fc312344f1e2d944f0aaecb637625752b2c2c47e039a/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e2f5dfa8e3ab7684082e8f772adcc173f1ccda51585c2c88247e56d3eb85bfd4/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/a44d60e53ab671bfaa1ac49cfd048b2fa10e6c45aa7bbcfa41ad25644326ec8d/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/752e5f1a15c84ebb98b4adc7d4d6ff2c26ce8629847b060747577f48d95d3d86/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/7188182cc77a497d15d59309ef732831cd6f73dcac805473dfbad5f3f5f32f06/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c38d5907274baf0e1160644aa9931d23c078810e100562e4131a7120bafa2345/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/fc39cd13d5cf374195032e8b38b101782a8049d3427de12cde3aa13738e5e384/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a062bb4cacfe91d88c2cf88dbabe30be54064fbae64d04a172535384534bfae4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/369a8a682086e64c240092eae1cd08548c9c1fd35654a6401bb6ad472b07145b/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/524cad2574e30cce568a0a8db4983211230c33e42fc5334b0210a59341cf51a7/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/6090fd38eceef923a86992082200e3511a2cbc8968848ce1b845bc9e60e1eb30/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/bac85e7c72af5ed069bb2f760d147ae4c53699840cfc3b89655606bc1da3903e/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ad6ba7f35dd37b4225c90687d875aca91daad76d38ff90339ad84f3f56fac570/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0730a2bfcdec9a00387f726dc683b438c1824b46d0819ae2e3e76c213ff9b026/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ee369bc6484c24f3695285deb1412e2aae4986b91473dea66b53abd7c1813abe/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c39cddf42ad32a842c3b52c946731359bacced065ef327b9b344fc176fc502fd/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3ac11a72f2fdc37a826b1cbde1fbc039fdf6cdbfb6199243b3ceff819c4a9860/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/21332c1b8c5e78214a2fdb2fad666b4ba40e31fe2b9eab4ccb56a4f5526406bb/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/81db1ea7ddd973e2cc6de1a975a35ba8236e7dff45cdb490fcf262d33007b215/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/1537c8163ff88a0a676ac7b3e81f7388b5fdd7ff79a0f465e6fd040625594518/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/fdfebe50ad2c150d15c0feb46ae40b4dfc9e2e783208381f86f550f48762ef25/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5f5942ecd4c66141667155b68c636c8d1f24622aece8a3eeda04b5209edae99b/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.769799 4934 manager.go:217] Machine: {Timestamp:2026-10-31 03:19:26.769345874 +0000 UTC m=+0.088088493 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105992 MemoryCapacity:32811995136 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6c5c2aa45d595ad30523d4f5a443 SystemUUID:ec2a6c5c-2aa4-5d59-5ad3-0523d4f5a443 BootID:97aefef7-4dfa-4de7-a6d7-2ee88f7db9d7 Filesystems:[{Device:/var/lib/kubelet/pods/c6c807d9-708e-4527-8e56-11749a26ca61/volumes/kubernetes.io~projected/kube-api-access-phhzp DeviceMajor:0 DeviceMinor:58 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/9ec0a926-6081-4a94-916a-2e27400b06a9/volumes/kubernetes.io~projected/kube-api-access-trx4z DeviceMajor:0 DeviceMinor:65 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~projected/kube-api-access-58zth DeviceMajor:0 DeviceMinor:226 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/1c5fbb4c-6297-4655-8c67-ff336a28e948/volumes/kubernetes.io~projected/kube-api-access-74zsl DeviceMajor:0 DeviceMinor:241 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19a9a3b840f5998529e86e283fa2ea71831b50665e81d68008d91f0b4897fca9/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c3b90117d898e769619cdec74f18d1e69f25206f4b6d63d100b8c6e5dd29e59/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/3a5ed6d1-a843-4d21-b913-59b0f9221e42/volumes/kubernetes.io~projected/kube-api-access-8ww68 DeviceMajor:0 DeviceMinor:216 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~projected/kube-api-access-jmczk DeviceMajor:0 DeviceMinor:221 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volumes/kubernetes.io~projected/kube-api-access-vzp8s DeviceMajor:0 DeviceMinor:57 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/437e8d737b8b666cb4aceb772a4327c2d99f84bbe2721c1799c57d7e1ce90a81/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15c8c5bb-10f1-4894-b45d-34b2209f9676/volumes/kubernetes.io~projected/kube-api-access-nnfr7 DeviceMajor:0 DeviceMinor:45 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe6c8824f78c4b17885dfe6daaf9774b4f1bc7a68260cb6bfed3971a24e588dc/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86348cd6e950a49f812da1f9212057a5cc992ab5d5cbc68b6bf780a54031b307/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b1ba02d33e619ce2f9882279d307b46171da0e12c021ffce8557a14b78916a6/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77907b880c28799f8c97c0150d5ca25dbbbe3c1534e2e5cbb8eaed6ed712c17c/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3aed8bd2-021f-4e33-96f0-e9c974fa0701/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1e1ea2182e46ce3b256ed3cc00bb75a1345a7cab033c2ce5cb8689a76cc3b10/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcd73e7ab4ad36809a42fd963afd3fbd9d2a4ccd86d02336ca45de5495430799/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/69dbc1e2-7632-40e1-8e81-79cf08b42d67/volumes/kubernetes.io~projected/kube-api-access-jxtgd DeviceMajor:0 DeviceMinor:197 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dba80bdeb3e79704bc274388bf14fccf7d5b4981c72c4d0f2c95b5505c96e3a3/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53e025546c07d72f5d9533e39a5f13c0fba1524ee57f8f3f0d8de768ae7b67f2/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/9bdf0731-bef7-4c9c-a5c3-09eff5aef394/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/2901d41f-f26c-4e6b-909f-1fc449e11ce8/volumes/kubernetes.io~projected/kube-api-access-s8xtp DeviceMajor:0 DeviceMinor:217 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405995520 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c3ac7e10b8a12eea0997eefb37a76c9fdd4453eb1a87cc0077a9d620241c91e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2/volumes/kubernetes.io~projected/kube-api-access-zqpm8 DeviceMajor:0 DeviceMinor:213 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/9f6df5b8-3430-46ee-8af1-13bb15bda816/volumes/kubernetes.io~projected/kube-api-access-xs2ps DeviceMajor:0 DeviceMinor:222 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e293a71db3bd5f844f1fe493ed4af34894e79aaf3abd77b346b8fc75dedbd983/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38b5a4b1d9e003591d385e70a51f3306a15cae63ce0f9be186bced90abe010f3/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~projected/kube-api-access-n27js DeviceMajor:0 DeviceMinor:214 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdd9600b-d5c9-4add-ae37-48aca3854510/volumes/kubernetes.io~projected/kube-api-access-z7scj DeviceMajor:0 DeviceMinor:56 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/1687ced2-8f63-4900-9b49-275a8104e159/volumes/kubernetes.io~projected/kube-api-access-rsdh2 DeviceMajor:0 DeviceMinor:64 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f6df5b8-3430-46ee-8af1-13bb15bda816/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3aed8bd2-021f-4e33-96f0-e9c974fa0701/volumes/kubernetes.io~projected/kube-api-access-2n2t5 DeviceMajor:0 DeviceMinor:224 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2e6ccf3cf471d62cd75c792e3ef1f94f53c4af0f747896364dcf8a8d962f6e2/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7befa522b24d9bfa6b0232bb97c32b94861ad61222826e8fa39ad6abc0dd4807/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/ad09c972-be06-48b1-84b0-a25fc1c645ba/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/283b8014-5ffd-4671-a28d-2975376d9cf9/volumes/kubernetes.io~projected/kube-api-access-kkjmw DeviceMajor:0 DeviceMinor:215 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e050b4-e196-498c-bb05-3b1bc20f10ae/volumes/kubernetes.io~projected/kube-api-access-l4tm6 DeviceMajor:0 DeviceMinor:335 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/b39f2c1f-e94d-4a91-aabd-08b7094aaa38/volumes/kubernetes.io~projected/kube-api-access-xzpcf DeviceMajor:0 DeviceMinor:66 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf4f6953-b84a-482d-b685-bd2addcbafde/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9bdf0731-bef7-4c9c-a5c3-09eff5aef394/volumes/kubernetes.io~projected/kube-api-access-xvlc5 DeviceMajor:0 DeviceMinor:219 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d/volumes/kubernetes.io~projected/kube-api-access-25l6m DeviceMajor:0 DeviceMinor:288 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c22d8d4598865ed32963a35212cc2a10673b9d9fbb9984199c6adf4cd6cdaaf3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90/volumes/kubernetes.io~projected/kube-api-access-5xj84 DeviceMajor:0 DeviceMinor:239 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/a57d414f-4fb4-4b78-92c7-105e2ac19d21/volumes/kubernetes.io~projected/kube-api-access-xfsgb DeviceMajor:0 DeviceMinor:218 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/6c58fbd8-7809-4718-8137-3439ac40c593/volumes/kubernetes.io~projected/kube-api-access-j8nt2 DeviceMajor:0 DeviceMinor:328 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eefb160-a377-431f-be0a-c2bea6ecb352/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bffa4ab83c06e166284c2b70616242842d5edae65e73d11961aefbfdd1cf7c7/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5419e24989d40d3a07f438241f3aab44261dc54a0f4a379a2a1c2e63daa8fa1/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16405999616 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/93cfa49d-0426-4d23-ad84-50f12272a871/volumes/kubernetes.io~projected/kube-api-access-sppr9 DeviceMajor:0 DeviceMinor:63 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a57d414f-4fb4-4b78-92c7-105e2ac19d21/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6/volumes/kubernetes.io~projected/kube-api-access-w8jgl DeviceMajor:0 DeviceMinor:220 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6c13c0-234d-4042-a31c-ac2c374fd1b3/volumes/kubernetes.io~projected/kube-api-access-7mqds DeviceMajor:0 DeviceMinor:62 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:/var/lib/kubelet/pods/3cfceaec-d546-4c9e-ac02-e66eeeea477a/volumes/kubernetes.io~projected/kube-api-access-qlwvt DeviceMajor:0 DeviceMinor:67 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52b0684eb07236bb6cbe33e42f936e5aead498ee528359dc2d37bf332b6d746d/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633395712 Type:vfs Inodes:4005370 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53ec70cb09a8678c05ca51e58b777087c00111deebef8f899c008bc2d7fdb68e/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005370 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:19a9a3b840f5998 MacAddress:12:08:d8:01:e0:cd Speed:10000 Mtu:8901} {Name:38b5a4b1d9e0035 MacAddress:22:0f:f3:30:66:a6 Speed:10000 Mtu:8901} {Name:3b1ba02d33e619c MacAddress:26:8f:e9:35:a9:c9 Speed:10000 Mtu:8901} {Name:52b0684eb07236b MacAddress:4a:2a:db:ba:5e:6c Speed:10000 Mtu:8901} {Name:53ec70cb09a8678 MacAddress:da:d3:ef:2f:10:29 Speed:10000 Mtu:8901} {Name:7befa522b24d9bf MacAddress:46:b8:72:c6:03:6d Speed:10000 Mtu:8901} {Name:7c3b90117d898e7 MacAddress:d2:64:e7:72:0d:c1 Speed:10000 Mtu:8901} {Name:86348cd6e950a49 MacAddress:fa:2d:80:5d:ea:3d Speed:10000 Mtu:8901} {Name:8bffa4ab83c06e1 MacAddress:ae:f0:c0:a7:93:22 Speed:10000 Mtu:8901} {Name:a1e1ea2182e46ce MacAddress:1a:0d:e0:0b:f4:6c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:69:2b:0a:db:1b Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:4c:f6:dc:3a:29 Speed:0 Mtu:8901} {Name:d2e6ccf3cf471d6 MacAddress:52:2d:ec:79:84:9b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:69:2b:0a:db:1b Speed:0 Mtu:9001} {Name:f5419e24989d40d MacAddress:0e:32:9c:53:25:cf Speed:10000 Mtu:8901} {Name:fe6c8824f78c4b1 MacAddress:3a:8e:8e:39:e3:d6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:8a:e7:64:a2:38:e0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:53:0b:ed:f7:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32811995136 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770196 4934 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770258 4934 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770548 4934 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770579 4934 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-238.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770701 4934 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770709 4934 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770719 4934 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:19:26.770781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770738 4934 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:19:26.771154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770829 4934 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:19:26.771154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.770999 4934 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:19:26.771154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.771116 4934 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:19:26.771154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.771153 4934 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:19:26.771287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.771169 4934 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:19:26.771287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.771178 4934 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:19:26.771287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.771187 4934 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:19:26.771287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.771195 4934 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:19:26.772096 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.772081 4934 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:19:26.772137 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.772102 4934 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:19:26.772596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.772579 4934 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:19:26.774786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.774767 4934 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:19:26.774786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.774783 4934 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:19:26.774934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.774892 4934 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 03:19:26.775125 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775109 4934 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:19:26.775189 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775151 4934 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:19:26.775298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775287 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:19:26.775353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775306 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:19:26.775353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775317 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:19:26.775353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775326 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:19:26.775353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775335 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:19:26.775353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775344 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:19:26.775353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775353 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:19:26.775625 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775362 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:19:26.775625 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775373 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:19:26.775625 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775383 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:19:26.775625 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775396 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:19:26.775625 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775422 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:19:26.775625 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775465 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:19:26.775625 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.775476 4934 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:19:26.778966 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.778950 4934 server.go:1263] "Started kubelet" Oct 31 03:19:26.779074 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.779044 4934 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:19:26.779126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.779051 4934 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:19:26.779170 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.779144 4934 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:19:26.779492 ip-10-0-138-238 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:19:26.779696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.779679 4934 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:19:26.780727 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.780710 4934 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:19:26.786781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.786756 4934 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:19:26.786888 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.786776 4934 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:19:26.788551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.786996 4934 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:19:26.788551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.787015 4934 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:19:26.788551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.787171 4934 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:19:26.788551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.787370 4934 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:47 +0000 UTC" deadline="2028-06-28 03:48:03.062526112 +0000 UTC" Oct 31 03:19:26.788551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.787446 4934 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14544h28m36.275087028s" Oct 31 03:19:26.788551 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:26.787021 4934 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:19:26.791462 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791424 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5fbb4c-6297-4655-8c67-ff336a28e948" volumeName="kubernetes.io/projected/1c5fbb4c-6297-4655-8c67-ff336a28e948-kube-api-access-74zsl" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791466 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" volumeName="kubernetes.io/secret/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-serving-cert" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791482 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6df5b8-3430-46ee-8af1-13bb15bda816" volumeName="kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-service-ca-bundle" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791493 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6df5b8-3430-46ee-8af1-13bb15bda816" volumeName="kubernetes.io/secret/9f6df5b8-3430-46ee-8af1-13bb15bda816-serving-cert" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791505 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4f6953-b84a-482d-b685-bd2addcbafde" volumeName="kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-env-overrides" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791516 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1687ced2-8f63-4900-9b49-275a8104e159" volumeName="kubernetes.io/projected/1687ced2-8f63-4900-9b49-275a8104e159-kube-api-access-rsdh2" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791527 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283b8014-5ffd-4671-a28d-2975376d9cf9" volumeName="kubernetes.io/projected/283b8014-5ffd-4671-a28d-2975376d9cf9-kube-api-access-kkjmw" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791539 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df315ba-3f9f-4566-a1c0-d969f1430072" volumeName="kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791552 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c58fbd8-7809-4718-8137-3439ac40c593" volumeName="kubernetes.io/empty-dir/6c58fbd8-7809-4718-8137-3439ac40c593-data-volume" seLinuxMountContext="" Oct 31 03:19:26.791565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791563 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6df5b8-3430-46ee-8af1-13bb15bda816" volumeName="kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-tmp" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791576 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6df5b8-3430-46ee-8af1-13bb15bda816" volumeName="kubernetes.io/projected/9f6df5b8-3430-46ee-8af1-13bb15bda816-kube-api-access-xs2ps" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791587 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6c13c0-234d-4042-a31c-ac2c374fd1b3" volumeName="kubernetes.io/configmap/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-serviceca" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791599 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3675be1-c31d-423f-83df-295df033f31a" volumeName="kubernetes.io/configmap/c3675be1-c31d-423f-83df-295df033f31a-nginx-conf" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791617 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1687ced2-8f63-4900-9b49-275a8104e159" volumeName="kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791632 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93cfa49d-0426-4d23-ad84-50f12272a871" volumeName="kubernetes.io/projected/93cfa49d-0426-4d23-ad84-50f12272a871-kube-api-access-sppr9" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791643 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdd9600b-d5c9-4add-ae37-48aca3854510" volumeName="kubernetes.io/configmap/bdd9600b-d5c9-4add-ae37-48aca3854510-iptables-alerter-script" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791655 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1687ced2-8f63-4900-9b49-275a8104e159" volumeName="kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791667 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90" volumeName="kubernetes.io/secret/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-cert" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791679 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cfceaec-d546-4c9e-ac02-e66eeeea477a" volumeName="kubernetes.io/projected/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kube-api-access-qlwvt" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791695 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad09c972-be06-48b1-84b0-a25fc1c645ba" volumeName="kubernetes.io/configmap/ad09c972-be06-48b1-84b0-a25fc1c645ba-konnectivity-ca" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791706 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4f6953-b84a-482d-b685-bd2addcbafde" volumeName="kubernetes.io/secret/cf4f6953-b84a-482d-b685-bd2addcbafde-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791718 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df315ba-3f9f-4566-a1c0-d969f1430072" volumeName="kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791730 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df315ba-3f9f-4566-a1c0-d969f1430072" volumeName="kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791741 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" volumeName="kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791759 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a5ed6d1-a843-4d21-b913-59b0f9221e42" volumeName="kubernetes.io/projected/3a5ed6d1-a843-4d21-b913-59b0f9221e42-kube-api-access-8ww68" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791771 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3aed8bd2-021f-4e33-96f0-e9c974fa0701" volumeName="kubernetes.io/projected/3aed8bd2-021f-4e33-96f0-e9c974fa0701-kube-api-access-2n2t5" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791794 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df315ba-3f9f-4566-a1c0-d969f1430072" volumeName="kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791807 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69dbc1e2-7632-40e1-8e81-79cf08b42d67" volumeName="kubernetes.io/projected/69dbc1e2-7632-40e1-8e81-79cf08b42d67-kube-api-access-jxtgd" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791833 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c58fbd8-7809-4718-8137-3439ac40c593" volumeName="kubernetes.io/projected/6c58fbd8-7809-4718-8137-3439ac40c593-kube-api-access-j8nt2" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791845 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" volumeName="kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-config" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791857 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" volumeName="kubernetes.io/projected/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-kube-api-access-xvlc5" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791868 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a57d414f-4fb4-4b78-92c7-105e2ac19d21" volumeName="kubernetes.io/projected/a57d414f-4fb4-4b78-92c7-105e2ac19d21-kube-api-access-xfsgb" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791880 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df315ba-3f9f-4566-a1c0-d969f1430072" volumeName="kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791891 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93cfa49d-0426-4d23-ad84-50f12272a871" volumeName="kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-cni-binary-copy" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791902 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" volumeName="kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791914 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" volumeName="kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791926 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" volumeName="kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791938 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c807d9-708e-4527-8e56-11749a26ca61" volumeName="kubernetes.io/empty-dir/c6c807d9-708e-4527-8e56-11749a26ca61-tmp-dir" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791950 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5fbb4c-6297-4655-8c67-ff336a28e948" volumeName="kubernetes.io/empty-dir/1c5fbb4c-6297-4655-8c67-ff336a28e948-tmp-dir" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791962 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283b8014-5ffd-4671-a28d-2975376d9cf9" volumeName="kubernetes.io/configmap/283b8014-5ffd-4671-a28d-2975376d9cf9-telemetry-config" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791975 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad09c972-be06-48b1-84b0-a25fc1c645ba" volumeName="kubernetes.io/secret/ad09c972-be06-48b1-84b0-a25fc1c645ba-agent-certs" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791987 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4f6953-b84a-482d-b685-bd2addcbafde" volumeName="kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-script-lib" seLinuxMountContext="" Oct 31 03:19:26.791988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.791999 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3aed8bd2-021f-4e33-96f0-e9c974fa0701" volumeName="kubernetes.io/secret/3aed8bd2-021f-4e33-96f0-e9c974fa0701-serving-cert" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792011 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c86d4eb-4dbf-4daf-95a3-62d73fc5210b" volumeName="kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-stats-auth" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792022 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdd9600b-d5c9-4add-ae37-48aca3854510" volumeName="kubernetes.io/projected/bdd9600b-d5c9-4add-ae37-48aca3854510-kube-api-access-z7scj" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792034 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2901d41f-f26c-4e6b-909f-1fc449e11ce8" volumeName="kubernetes.io/projected/2901d41f-f26c-4e6b-909f-1fc449e11ce8-kube-api-access-s8xtp" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792047 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4f6953-b84a-482d-b685-bd2addcbafde" volumeName="kubernetes.io/projected/cf4f6953-b84a-482d-b685-bd2addcbafde-kube-api-access-vzp8s" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792059 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3aed8bd2-021f-4e33-96f0-e9c974fa0701" volumeName="kubernetes.io/configmap/3aed8bd2-021f-4e33-96f0-e9c974fa0701-config" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792070 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" volumeName="kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-trusted-ca" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792081 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39f2c1f-e94d-4a91-aabd-08b7094aaa38" volumeName="kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-tuned" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792093 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" volumeName="kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792105 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2" volumeName="kubernetes.io/projected/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-kube-api-access-zqpm8" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792118 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15c8c5bb-10f1-4894-b45d-34b2209f9676" volumeName="kubernetes.io/projected/15c8c5bb-10f1-4894-b45d-34b2209f9676-kube-api-access-nnfr7" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792130 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df315ba-3f9f-4566-a1c0-d969f1430072" volumeName="kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792142 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93cfa49d-0426-4d23-ad84-50f12272a871" volumeName="kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-multus-daemon-config" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792153 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a57d414f-4fb4-4b78-92c7-105e2ac19d21" volumeName="kubernetes.io/configmap/a57d414f-4fb4-4b78-92c7-105e2ac19d21-config" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792174 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39f2c1f-e94d-4a91-aabd-08b7094aaa38" volumeName="kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-tmp" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792186 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39f2c1f-e94d-4a91-aabd-08b7094aaa38" volumeName="kubernetes.io/projected/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-kube-api-access-xzpcf" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792198 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" volumeName="kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792209 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" volumeName="kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792220 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90" volumeName="kubernetes.io/projected/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-kube-api-access-5xj84" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792251 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ec0a926-6081-4a94-916a-2e27400b06a9" volumeName="kubernetes.io/projected/9ec0a926-6081-4a94-916a-2e27400b06a9-kube-api-access-trx4z" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792264 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2" volumeName="kubernetes.io/configmap/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-config" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792283 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e050b4-e196-498c-bb05-3b1bc20f10ae" volumeName="kubernetes.io/projected/d0e050b4-e196-498c-bb05-3b1bc20f10ae-kube-api-access-l4tm6" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792296 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cc36af9-eaa8-40a3-b642-cc7aaf970ed6" volumeName="kubernetes.io/projected/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6-kube-api-access-w8jgl" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792307 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df315ba-3f9f-4566-a1c0-d969f1430072" volumeName="kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792320 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c86d4eb-4dbf-4daf-95a3-62d73fc5210b" volumeName="kubernetes.io/projected/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-kube-api-access-jmczk" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792332 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c86d4eb-4dbf-4daf-95a3-62d73fc5210b" volumeName="kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-default-certificate" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792344 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6df5b8-3430-46ee-8af1-13bb15bda816" volumeName="kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-trusted-ca-bundle" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792356 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6df5b8-3430-46ee-8af1-13bb15bda816" volumeName="kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-snapshots" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792369 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5fbb4c-6297-4655-8c67-ff336a28e948" volumeName="kubernetes.io/configmap/1c5fbb4c-6297-4655-8c67-ff336a28e948-config-volume" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792381 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eefb160-a377-431f-be0a-c2bea6ecb352" volumeName="kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792392 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6c13c0-234d-4042-a31c-ac2c374fd1b3" volumeName="kubernetes.io/projected/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-kube-api-access-7mqds" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792404 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6c807d9-708e-4527-8e56-11749a26ca61" volumeName="kubernetes.io/projected/c6c807d9-708e-4527-8e56-11749a26ca61-kube-api-access-phhzp" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792416 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4f6953-b84a-482d-b685-bd2addcbafde" volumeName="kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-config" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792428 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f15c9c88-bfe3-4acd-8047-9c3d9cd0296d" volumeName="kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792442 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1687ced2-8f63-4900-9b49-275a8104e159" volumeName="kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-binary-copy" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792461 4934 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a57d414f-4fb4-4b78-92c7-105e2ac19d21" volumeName="kubernetes.io/secret/a57d414f-4fb4-4b78-92c7-105e2ac19d21-serving-cert" seLinuxMountContext="" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792472 4934 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:19:26.793174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.792479 4934 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:19:26.795478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795452 4934 factory.go:153] Registering CRI-O factory Oct 31 03:19:26.795478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795471 4934 factory.go:223] Registration of the crio container factory successfully Oct 31 03:19:26.795612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795528 4934 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:19:26.795612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795538 4934 factory.go:55] Registering systemd factory Oct 31 03:19:26.795612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795546 4934 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:19:26.795612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795562 4934 factory.go:103] Registering Raw factory Oct 31 03:19:26.795612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795577 4934 manager.go:1199] Started watching for new ooms in manager Oct 31 03:19:26.795959 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.795946 4934 manager.go:319] Starting recovery of all containers Oct 31 03:19:26.803470 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.803446 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/1.log" Oct 31 03:19:26.803755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.803736 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/1.log" Oct 31 03:19:26.812294 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.812252 4934 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:19:26.813923 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.813895 4934 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:19:26.814033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.813929 4934 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:19:26.814033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.813950 4934 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:19:26.814033 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:26.813991 4934 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 03:19:26.821694 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.821661 4934 generic.go:356] "Generic (PLEG): container finished" podID="87725f65687ad3eca42b0f7a2fe25aa4" containerID="9804aabb435e85f48b75047d13dd156bd7d3a2ff969d9ef2a88a94ae664b3a5e" exitCode=0 Oct 31 03:19:26.833967 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.833936 4934 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="5a8d9c7f7e41324bfb39823b52d39aa749e88a7dedf17087dfd372c074f6672a" exitCode=0 Oct 31 03:19:26.833967 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.833965 4934 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="8c6de07e13b9c354531cb9a954fcbaabe0757a7be68a0cb77ce347e12981e5da" exitCode=0 Oct 31 03:19:26.834101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.833975 4934 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="ef6f1f4e35fd764356ee664b82dae187be33c4f9d0a0e7bdd5c504af7de676fc" exitCode=0 Oct 31 03:19:26.834101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.833983 4934 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="ee05b01eae1ac67eadc2b9a67d2ea2998ad6ee9133ac43d2edac2cf46504aa91" exitCode=0 Oct 31 03:19:26.834101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.833994 4934 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="ad75098623277f700d60c008a37718a6da7cf0dda9e87a6773defbbe20dfb85f" exitCode=0 Oct 31 03:19:26.834101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.834000 4934 generic.go:356] "Generic (PLEG): container finished" podID="1687ced2-8f63-4900-9b49-275a8104e159" containerID="00b84820fc469e6bdbec4fbb0fa727f7abe8717062325efae61a8de72565d861" exitCode=0 Oct 31 03:19:26.863663 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.863632 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/1.log" Oct 31 03:19:26.863809 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.863672 4934 generic.go:356] "Generic (PLEG): container finished" podID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" containerID="94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4" exitCode=255 Oct 31 03:19:26.887747 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:26.887667 4934 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:19:26.888188 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.888166 4934 manager.go:324] Recovery completed Oct 31 03:19:26.905902 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.905876 4934 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:26.907150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.907131 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:26.907255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.907162 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:26.907255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.907179 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:26.909202 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909180 4934 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:19:26.909202 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909195 4934 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:19:26.909202 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909198 4934 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:26.909425 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909214 4934 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:19:26.909425 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909390 4934 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 03:19:26.909425 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909406 4934 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 03:19:26.909558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909432 4934 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 03:19:26.909558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909441 4934 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 03:19:26.909558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909449 4934 policy_none.go:50] "Start" Oct 31 03:19:26.909558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909461 4934 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:19:26.909558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909474 4934 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:19:26.909814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909693 4934 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 03:19:26.909814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909705 4934 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 03:19:26.909814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.909714 4934 policy_none.go:44] "Start" Oct 31 03:19:26.910094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.910080 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:26.910094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.910104 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:26.910212 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.910114 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:26.914092 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.914065 4934 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:19:26.914174 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:26.914073 4934 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 03:19:26.919403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.919390 4934 manager.go:341] "Starting Device Plugin manager" Oct 31 03:19:26.919460 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:26.919415 4934 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:19:26.919460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.919425 4934 server.go:87] "Starting device plugin registration server" Oct 31 03:19:26.919739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.919728 4934 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:19:26.919772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.919741 4934 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:19:26.919871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.919852 4934 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:19:26.919995 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.919955 4934 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:19:26.920100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.920008 4934 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:19:26.920100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.920015 4934 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:19:26.920206 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.920079 4934 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:19:26.919991499Z","UUID":"7f17205b-ffa6-4b06-9c87-0ee7e04faddb","Handler":null,"Name":"","Endpoint":""} Oct 31 03:19:26.921726 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:26.921696 4934 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:19:26.921825 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:26.921752 4934 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-238.us-west-1.compute.internal\" not found" Oct 31 03:19:26.922184 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.922164 4934 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:19:26.922285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.922206 4934 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:19:26.983142 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.983110 4934 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:26.983142 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.983123 4934 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:26.983340 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.983270 4934 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 03:19:26.984135 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:26.984121 4934 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:27.020291 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.020267 4934 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:27.021763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.021739 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:27.021897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.021776 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:27.021897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.021787 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:27.021897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.021829 4934 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.034487 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.034466 4934 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:27.114397 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.114343 4934 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal"] Oct 31 03:19:27.114991 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.114952 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"87725f65687ad3eca42b0f7a2fe25aa4","Type":"ContainerDied","Data":"9804aabb435e85f48b75047d13dd156bd7d3a2ff969d9ef2a88a94ae664b3a5e"} Oct 31 03:19:27.115101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.114999 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"87725f65687ad3eca42b0f7a2fe25aa4","Type":"ContainerStarted","Data":"16713b8b2dbe41e22694d194fd1d282099e6fda1b7315b36bc4425a6d10126b8"} Oct 31 03:19:27.115101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.115051 4934 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.115101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.115077 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"0de19d3008b27f6bbf6579b9645a2ecb","Type":"ContainerStarted","Data":"da286635cdd55b852f86e81ee02173ce9a832cab14e01d8bb6299821a76c1251"} Oct 31 03:19:27.115101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.115096 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" event={"ID":"0de19d3008b27f6bbf6579b9645a2ecb","Type":"ContainerStarted","Data":"01f0c60f52dbe4c07a5ec44bd8da656b1de30ef9c99e704d07481a51add903f4"} Oct 31 03:19:27.115302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.115212 4934 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.139063 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.138999 4934 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.139176 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.139068 4934 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.139673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.139658 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeNotReady" Oct 31 03:19:27.139727 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.139688 4934 setters.go:546] "Node became not ready" node="ip-10-0-138-238.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T03:19:27Z","lastTransitionTime":"2026-10-31T03:19:27Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 03:19:27.148247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.148204 4934 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-238.us-west-1.compute.internal" is owned by different node (oldNodeID="548c2cbf-74f3-4f57-b15c-e9ff1b230a61", newNodeID=""), cleaning up... Oct 31 03:19:27.178407 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.178389 4934 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:27.178512 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:27.178444 4934 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.178961 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.178948 4934 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:27.179027 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:27.178982 4934 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.284889 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.284858 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.284889 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.284891 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.285081 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.284909 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de19d3008b27f6bbf6579b9645a2ecb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"0de19d3008b27f6bbf6579b9645a2ecb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.385483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.385447 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.385483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.385482 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.385681 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.385505 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de19d3008b27f6bbf6579b9645a2ecb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"0de19d3008b27f6bbf6579b9645a2ecb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.385681 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.385554 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0de19d3008b27f6bbf6579b9645a2ecb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"0de19d3008b27f6bbf6579b9645a2ecb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.385681 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.385568 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.385681 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.385554 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87725f65687ad3eca42b0f7a2fe25aa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" (UID: \"87725f65687ad3eca42b0f7a2fe25aa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.772243 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.772188 4934 apiserver.go:52] "Watching apiserver" Oct 31 03:19:27.872317 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.872289 4934 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.872453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.872419 4934 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.926065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.926024 4934 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:19:27.926820 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.926794 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-wdw6h","openshift-ovn-kubernetes/ovnkube-node-gwqdn","openshift-network-console/networking-console-plugin-86c478bb88-xbfpv","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks","openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx","openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww","openshift-dns/dns-default-4k5qr","kube-system/kas-connection-checker-685895c7c9-pkgt6","kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb","openshift-console-operator/console-operator-7cd56b8dbc-b54xh","kube-system/global-pull-secret-syncer-5ngpx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal","openshift-image-registry/node-ca-ntzrg","openshift-multus/network-metrics-daemon-z2hw8","kube-system/konnectivity-agent-vgq6v","openshift-network-operator/iptables-alerter-qgw8p","openshift-insights/insights-runtime-extractor-z7s46","openshift-image-registry/image-registry-79f9f4c5f5-j4qrr","openshift-cluster-node-tuning-operator/tuned-hhs74","openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw","openshift-ingress/router-default-6d69c76ccf-dccpx","openshift-dns/node-resolver-kh444","openshift-multus/multus-additional-cni-plugins-2r4ff","openshift-image-registry/image-registry-58d7548854-gr79z","openshift-multus/multus-jw6nt","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h","openshift-ingress-canary/ingress-canary-pvphc","kube-system/kas-connection-checker-685895c7c9-mdl5l","openshift-insights/insights-operator-7c4986d449-zrtsh","kube-system/kas-connection-checker-685895c7c9-mdl5v"] Oct 31 03:19:27.928373 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.928344 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:27.932180 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.930824 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:27.932811 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.932784 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:27.933261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.933220 4934 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:27.933369 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:27.933297 4934 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.933675 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.933659 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:27.933909 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.933886 4934 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:27.933957 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:27.933943 4934 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-238.us-west-1.compute.internal" Oct 31 03:19:27.934484 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.934468 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:27.936926 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.936910 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:27.941177 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.941158 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:27.942407 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.942393 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:27.942950 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.942938 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:27.979943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.979912 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 03:19:27.979943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.979912 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 03:19:27.980167 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980150 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 03:19:27.980209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980195 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:27.980362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980341 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 03:19:27.980405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980395 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 03:19:27.980451 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980442 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8zl9c\"" Oct 31 03:19:27.980846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980832 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 03:19:27.980891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980877 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-q872x\"" Oct 31 03:19:27.980985 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980971 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 03:19:27.981025 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.980995 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:19:27.981062 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981022 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-242zg\"" Oct 31 03:19:27.981159 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981145 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:19:27.981206 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981159 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:19:27.981206 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981165 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:27.981469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981451 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 03:19:27.981469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981475 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:27.981612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981581 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tgrsk\"" Oct 31 03:19:27.981771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981757 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 03:19:27.981834 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.981776 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:27.982401 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.982385 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:27.989640 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:27.989627 4934 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:19:28.036693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.036628 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.037324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037287 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 03:19:28.037324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037303 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 03:19:28.037507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037366 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mnxkk\"" Oct 31 03:19:28.037507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037448 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.037507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037475 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.037507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037504 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 03:19:28.037671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037544 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.037964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.037947 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.038116 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.038081 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 03:19:28.038260 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.038023 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 03:19:28.038469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.038453 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 03:19:28.038642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.038464 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.038786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.038467 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:19:28.038866 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.038847 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.038979 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.038833 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 03:19:28.039177 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.039131 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hhcvs\"" Oct 31 03:19:28.039177 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.039172 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.039351 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.039217 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 03:19:28.039351 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.039333 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.039487 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.039020 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 03:19:28.039487 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.039431 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jjf7q\"" Oct 31 03:19:28.039581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.039522 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 03:19:28.040573 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.040549 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:19:28.081214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081186 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:19:28.081364 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081280 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 03:19:28.081364 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081289 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:19:28.081752 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081681 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.081857 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081798 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkxjs\"" Oct 31 03:19:28.081857 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081800 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.081857 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081841 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9x6rv\"" Oct 31 03:19:28.081980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081864 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-n6jvn\"" Oct 31 03:19:28.081980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081917 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:19:28.081980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.081974 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.082134 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082120 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.082188 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082174 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 03:19:28.082267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082181 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:19:28.082267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082201 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 03:19:28.082267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082214 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082280 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082285 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082297 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082323 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082354 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rnwzl\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082360 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082366 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.082411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082385 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xz7gs\"" Oct 31 03:19:28.082679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082450 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.082679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082458 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sv7k2\"" Oct 31 03:19:28.082679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082463 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:19:28.082679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082469 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:19:28.082679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082480 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.082679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082520 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.082679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082559 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082779 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082823 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082845 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082874 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082902 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b5nvx\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082911 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082912 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67ksm\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082927 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fn2lp\"" Oct 31 03:19:28.082929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082904 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btzr7\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082937 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dbgqm\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082946 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082953 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082974 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v5ddh\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.082983 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083013 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083041 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083050 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7qxg9\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083065 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vxflf\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083075 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083080 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083093 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083112 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083138 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltpjd\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083142 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4ln2\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083147 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4wd7m\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083165 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4kwqf\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083187 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083188 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083204 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083222 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083286 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083286 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xmnbh\"" Oct 31 03:19:28.083343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.083337 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:19:28.088663 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.088647 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-etc-kubernetes\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.088703 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.088674 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-cni-binary-copy\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.088703 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.088690 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-socket-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.088774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.088708 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zsl\" (UniqueName: \"kubernetes.io/projected/1c5fbb4c-6297-4655-8c67-ff336a28e948-kube-api-access-74zsl\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.088774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.088732 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtgd\" (UniqueName: \"kubernetes.io/projected/69dbc1e2-7632-40e1-8e81-79cf08b42d67-kube-api-access-jxtgd\") pod \"kas-connection-checker-685895c7c9-mdl5v\" (UID: \"69dbc1e2-7632-40e1-8e81-79cf08b42d67\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" Oct 31 03:19:28.088982 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.088966 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlc5\" (UniqueName: \"kubernetes.io/projected/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-kube-api-access-xvlc5\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.089020 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.088997 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c58fbd8-7809-4718-8137-3439ac40c593-data-volume\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.089020 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089005 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-cni-binary-copy\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.089098 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089024 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-etc-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.089098 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089060 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-serviceca\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:19:28.089191 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089133 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trx4z\" (UniqueName: \"kubernetes.io/projected/9ec0a926-6081-4a94-916a-2e27400b06a9-kube-api-access-trx4z\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:28.089191 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089139 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c58fbd8-7809-4718-8137-3439ac40c593-data-volume\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.089191 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089173 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.089319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089199 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdd9600b-d5c9-4add-ae37-48aca3854510-iptables-alerter-script\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:19:28.089319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089222 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7scj\" (UniqueName: \"kubernetes.io/projected/bdd9600b-d5c9-4add-ae37-48aca3854510-kube-api-access-z7scj\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:19:28.089319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089269 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aed8bd2-021f-4e33-96f0-e9c974fa0701-serving-cert\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:28.089319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089295 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.089319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089305 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-serviceca\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:19:28.089537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089323 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.089537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089373 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:28.089537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089405 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjmw\" (UniqueName: \"kubernetes.io/projected/283b8014-5ffd-4671-a28d-2975376d9cf9-kube-api-access-kkjmw\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:28.089537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089434 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.089537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089466 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.089537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089490 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aed8bd2-021f-4e33-96f0-e9c974fa0701-serving-cert\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:28.089537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089496 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-kubelet\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089558 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdd9600b-d5c9-4add-ae37-48aca3854510-iptables-alerter-script\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089566 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf4f6953-b84a-482d-b685-bd2addcbafde-ovn-node-metrics-cert\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089614 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089627 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089665 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-serving-cert\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089787 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-default-certificate\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089815 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf4f6953-b84a-482d-b685-bd2addcbafde-ovn-node-metrics-cert\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.089847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089843 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089882 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089920 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089927 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089925 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-serving-cert\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.089972 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-device-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090010 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5fbb4c-6297-4655-8c67-ff336a28e948-tmp-dir\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090026 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-default-certificate\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090047 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090086 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-host\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090096 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5fbb4c-6297-4655-8c67-ff336a28e948-tmp-dir\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090111 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090111 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-cnibin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090142 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090148 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-snapshots\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090181 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-ovn\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.090213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090199 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-node-log\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090288 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-system-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090325 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090351 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-bin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090382 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090386 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-socket-dir-parent\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090387 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-snapshots\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090428 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-hostroot\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090455 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-conf-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090491 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-netns\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090518 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zth\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090538 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdh2\" (UniqueName: \"kubernetes.io/projected/1687ced2-8f63-4900-9b49-275a8104e159-kube-api-access-rsdh2\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.090646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090573 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad09c972-be06-48b1-84b0-a25fc1c645ba-konnectivity-ca\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090666 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090760 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-conf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090831 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090858 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2t5\" (UniqueName: \"kubernetes.io/projected/3aed8bd2-021f-4e33-96f0-e9c974fa0701-kube-api-access-2n2t5\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090867 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad09c972-be06-48b1-84b0-a25fc1c645ba-konnectivity-ca\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090910 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090943 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-system-cni-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.090994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.090969 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysconfig\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.091219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091051 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzp8s\" (UniqueName: \"kubernetes.io/projected/cf4f6953-b84a-482d-b685-bd2addcbafde-kube-api-access-vzp8s\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.091219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091095 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-tmp\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.091219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091143 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c3675be1-c31d-423f-83df-295df033f31a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:28.091219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091174 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c807d9-708e-4527-8e56-11749a26ca61-tmp-dir\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:19:28.091219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091203 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-env-overrides\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.091219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091213 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-tmp\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091250 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-var-lib-kubelet\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091263 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091287 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-multus\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091304 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c807d9-708e-4527-8e56-11749a26ca61-tmp-dir\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091326 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-sys-fs\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091337 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c3675be1-c31d-423f-83df-295df033f31a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091372 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/283b8014-5ffd-4671-a28d-2975376d9cf9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091397 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-env-overrides\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091412 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-log-socket\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091437 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/15c8c5bb-10f1-4894-b45d-34b2209f9676-kube-api-access-nnfr7\") pod \"kas-connection-checker-685895c7c9-mdl5l\" (UID: \"15c8c5bb-10f1-4894-b45d-34b2209f9676\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" Oct 31 03:19:28.091510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091465 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091569 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091606 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6df5b8-3430-46ee-8af1-13bb15bda816-serving-cert\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091627 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-kubernetes\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091691 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-systemd\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091706 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3eefb160-a377-431f-be0a-c2bea6ecb352-original-pull-secret\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091711 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/283b8014-5ffd-4671-a28d-2975376d9cf9-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091720 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4tm6\" (UniqueName: \"kubernetes.io/projected/d0e050b4-e196-498c-bb05-3b1bc20f10ae-kube-api-access-l4tm6\") pod \"migrator-66df5d496b-nv8wr\" (UID: \"d0e050b4-e196-498c-bb05-3b1bc20f10ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091736 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6df5b8-3430-46ee-8af1-13bb15bda816-serving-cert\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091744 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xj84\" (UniqueName: \"kubernetes.io/projected/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-kube-api-access-5xj84\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091771 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.091921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091790 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.092203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.091934 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpm8\" (UniqueName: \"kubernetes.io/projected/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-kube-api-access-zqpm8\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:28.092203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092008 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2ps\" (UniqueName: \"kubernetes.io/projected/9f6df5b8-3430-46ee-8af1-13bb15bda816-kube-api-access-xs2ps\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.092203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092090 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.092203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092137 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27js\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.092203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092167 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6c807d9-708e-4527-8e56-11749a26ca61-hosts-file\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:19:28.092362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092251 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhzp\" (UniqueName: \"kubernetes.io/projected/c6c807d9-708e-4527-8e56-11749a26ca61-kube-api-access-phhzp\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:19:28.092362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092334 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c58fbd8-7809-4718-8137-3439ac40c593-crio-socket\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.092362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092354 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-systemd-units\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.092443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092371 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-trusted-ca\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.092443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092387 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-k8s-cni-cncf-io\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.092521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092504 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-multus-daemon-config\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.092594 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092582 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mqds\" (UniqueName: \"kubernetes.io/projected/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-kube-api-access-7mqds\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:19:28.092673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092656 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-trusted-ca\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.092706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092669 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93cfa49d-0426-4d23-ad84-50f12272a871-multus-daemon-config\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.092706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092668 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a57d414f-4fb4-4b78-92c7-105e2ac19d21-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:28.092761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092714 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xtp\" (UniqueName: \"kubernetes.io/projected/2901d41f-f26c-4e6b-909f-1fc449e11ce8-kube-api-access-s8xtp\") pod \"network-check-source-6f78d59f99-t4nxw\" (UID: \"2901d41f-f26c-4e6b-909f-1fc449e11ce8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" Oct 31 03:19:28.092761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092741 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.092853 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092769 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpcf\" (UniqueName: \"kubernetes.io/projected/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-kube-api-access-xzpcf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.092853 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092833 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a57d414f-4fb4-4b78-92c7-105e2ac19d21-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:28.092853 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092835 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.092951 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092881 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-modprobe-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.092951 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092905 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-netns\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.093010 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092949 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1687ced2-8f63-4900-9b49-275a8104e159-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.093010 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092981 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-multus-certs\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.093010 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.092996 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.093088 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093012 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:28.093088 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093041 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-tmp\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.093088 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093060 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww68\" (UniqueName: \"kubernetes.io/projected/3a5ed6d1-a843-4d21-b913-59b0f9221e42-kube-api-access-8ww68\") pod \"volume-data-source-validator-58c9f5f46d-87lks\" (UID: \"3a5ed6d1-a843-4d21-b913-59b0f9221e42\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" Oct 31 03:19:28.093088 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093076 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-run\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.093211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093092 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-kubelet-config\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.093211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093124 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6df5b8-3430-46ee-8af1-13bb15bda816-tmp\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.093211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093139 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-script-lib\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.093211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093166 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-kubelet\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.093211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093192 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jgl\" (UniqueName: \"kubernetes.io/projected/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6-kube-api-access-w8jgl\") pod \"kas-connection-checker-685895c7c9-pkgt6\" (UID: \"4cc36af9-eaa8-40a3-b642-cc7aaf970ed6\") " pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" Oct 31 03:19:28.093211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093201 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093219 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-service-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093278 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a57d414f-4fb4-4b78-92c7-105e2ac19d21-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093323 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093345 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-script-lib\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093382 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093400 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6df5b8-3430-46ee-8af1-13bb15bda816-service-ca-bundle\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093424 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwvt\" (UniqueName: \"kubernetes.io/projected/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kube-api-access-qlwvt\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093448 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.093472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093464 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-var-lib-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093475 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a57d414f-4fb4-4b78-92c7-105e2ac19d21-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093503 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-sys\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093519 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-dbus\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093533 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-slash\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093561 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093600 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093629 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-tuned\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093684 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-tuned\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093702 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppr9\" (UniqueName: \"kubernetes.io/projected/93cfa49d-0426-4d23-ad84-50f12272a871-kube-api-access-sppr9\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093718 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-registration-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093734 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aed8bd2-021f-4e33-96f0-e9c974fa0701-config\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:28.093788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093769 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-config\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093827 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093846 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsgb\" (UniqueName: \"kubernetes.io/projected/a57d414f-4fb4-4b78-92c7-105e2ac19d21-kube-api-access-xfsgb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093865 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-bin\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093881 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-cnibin\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093896 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093956 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aed8bd2-021f-4e33-96f0-e9c974fa0701-config\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093962 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf4f6953-b84a-482d-b685-bd2addcbafde-ovnkube-config\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.093989 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094010 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-host\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094063 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094085 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094127 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094197 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094205 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094256 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nt2\" (UniqueName: \"kubernetes.io/projected/6c58fbd8-7809-4718-8137-3439ac40c593-kube-api-access-j8nt2\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094273 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-netd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094289 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdd9600b-d5c9-4add-ae37-48aca3854510-host-slash\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094312 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad09c972-be06-48b1-84b0-a25fc1c645ba-agent-certs\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094344 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.094361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094351 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmczk\" (UniqueName: \"kubernetes.io/projected/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-kube-api-access-jmczk\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094410 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-config\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094516 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094562 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad09c972-be06-48b1-84b0-a25fc1c645ba-agent-certs\") pod \"konnectivity-agent-vgq6v\" (UID: \"ad09c972-be06-48b1-84b0-a25fc1c645ba\") " pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094566 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094606 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094636 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094636 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-config\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094738 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094751 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-lib-modules\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094778 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-cert\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094846 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fbb4c-6297-4655-8c67-ff336a28e948-config-volume\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094876 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-stats-auth\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094902 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-systemd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094906 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-cert\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094929 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.094995 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.095064 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-stats-auth\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.095077 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.095065 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5fbb4c-6297-4655-8c67-ff336a28e948-config-volume\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.148679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.148638 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtgd\" (UniqueName: \"kubernetes.io/projected/69dbc1e2-7632-40e1-8e81-79cf08b42d67-kube-api-access-jxtgd\") pod \"kas-connection-checker-685895c7c9-mdl5v\" (UID: \"69dbc1e2-7632-40e1-8e81-79cf08b42d67\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5v" Oct 31 03:19:28.148827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.148704 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7scj\" (UniqueName: \"kubernetes.io/projected/bdd9600b-d5c9-4add-ae37-48aca3854510-kube-api-access-z7scj\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:19:28.148827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.148711 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zsl\" (UniqueName: \"kubernetes.io/projected/1c5fbb4c-6297-4655-8c67-ff336a28e948-kube-api-access-74zsl\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.148827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.148757 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trx4z\" (UniqueName: \"kubernetes.io/projected/9ec0a926-6081-4a94-916a-2e27400b06a9-kube-api-access-trx4z\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:28.148827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.148783 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlc5\" (UniqueName: \"kubernetes.io/projected/9bdf0731-bef7-4c9c-a5c3-09eff5aef394-kube-api-access-xvlc5\") pod \"console-operator-7cd56b8dbc-b54xh\" (UID: \"9bdf0731-bef7-4c9c-a5c3-09eff5aef394\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:28.149068 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.149052 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25l6m\" (UniqueName: \"kubernetes.io/projected/f15c9c88-bfe3-4acd-8047-9c3d9cd0296d-kube-api-access-25l6m\") pod \"network-check-target-wdw6h\" (UID: \"f15c9c88-bfe3-4acd-8047-9c3d9cd0296d\") " pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:28.153826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.153805 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/15c8c5bb-10f1-4894-b45d-34b2209f9676-kube-api-access-nnfr7\") pod \"kas-connection-checker-685895c7c9-mdl5l\" (UID: \"15c8c5bb-10f1-4894-b45d-34b2209f9676\") " pod="kube-system/kas-connection-checker-685895c7c9-mdl5l" Oct 31 03:19:28.155777 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.155760 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjmw\" (UniqueName: \"kubernetes.io/projected/283b8014-5ffd-4671-a28d-2975376d9cf9-kube-api-access-kkjmw\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:28.156686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.156659 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.156770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.156733 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2t5\" (UniqueName: \"kubernetes.io/projected/3aed8bd2-021f-4e33-96f0-e9c974fa0701-kube-api-access-2n2t5\") pod \"service-ca-operator-544d749c8c-jpxww\" (UID: \"3aed8bd2-021f-4e33-96f0-e9c974fa0701\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" Oct 31 03:19:28.156770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.156751 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzp8s\" (UniqueName: \"kubernetes.io/projected/cf4f6953-b84a-482d-b685-bd2addcbafde-kube-api-access-vzp8s\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.156845 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.156779 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zth\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.156845 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.156806 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdh2\" (UniqueName: \"kubernetes.io/projected/1687ced2-8f63-4900-9b49-275a8104e159-kube-api-access-rsdh2\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.159567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.159543 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwvt\" (UniqueName: \"kubernetes.io/projected/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kube-api-access-qlwvt\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.159648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.159588 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmczk\" (UniqueName: \"kubernetes.io/projected/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-kube-api-access-jmczk\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.160085 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.160065 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppr9\" (UniqueName: \"kubernetes.io/projected/93cfa49d-0426-4d23-ad84-50f12272a871-kube-api-access-sppr9\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.160120 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.160071 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpm8\" (UniqueName: \"kubernetes.io/projected/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-kube-api-access-zqpm8\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:28.160325 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.160311 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27js\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.160911 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.160887 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpcf\" (UniqueName: \"kubernetes.io/projected/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-kube-api-access-xzpcf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.161574 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.161555 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsgb\" (UniqueName: \"kubernetes.io/projected/a57d414f-4fb4-4b78-92c7-105e2ac19d21-kube-api-access-xfsgb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h\" (UID: \"a57d414f-4fb4-4b78-92c7-105e2ac19d21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" Oct 31 03:19:28.161619 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.161601 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4tm6\" (UniqueName: \"kubernetes.io/projected/d0e050b4-e196-498c-bb05-3b1bc20f10ae-kube-api-access-l4tm6\") pod \"migrator-66df5d496b-nv8wr\" (UID: \"d0e050b4-e196-498c-bb05-3b1bc20f10ae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nv8wr" Oct 31 03:19:28.161650 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.161601 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nt2\" (UniqueName: \"kubernetes.io/projected/6c58fbd8-7809-4718-8137-3439ac40c593-kube-api-access-j8nt2\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.161870 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.161855 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhzp\" (UniqueName: \"kubernetes.io/projected/c6c807d9-708e-4527-8e56-11749a26ca61-kube-api-access-phhzp\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:19:28.162156 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.162140 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2ps\" (UniqueName: \"kubernetes.io/projected/9f6df5b8-3430-46ee-8af1-13bb15bda816-kube-api-access-xs2ps\") pod \"insights-operator-7c4986d449-zrtsh\" (UID: \"9f6df5b8-3430-46ee-8af1-13bb15bda816\") " pod="openshift-insights/insights-operator-7c4986d449-zrtsh" Oct 31 03:19:28.162565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.162550 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xj84\" (UniqueName: \"kubernetes.io/projected/2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90-kube-api-access-5xj84\") pod \"ingress-canary-pvphc\" (UID: \"2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90\") " pod="openshift-ingress-canary/ingress-canary-pvphc" Oct 31 03:19:28.164453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.164433 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww68\" (UniqueName: \"kubernetes.io/projected/3a5ed6d1-a843-4d21-b913-59b0f9221e42-kube-api-access-8ww68\") pod \"volume-data-source-validator-58c9f5f46d-87lks\" (UID: \"3a5ed6d1-a843-4d21-b913-59b0f9221e42\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87lks" Oct 31 03:19:28.164750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.164733 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mqds\" (UniqueName: \"kubernetes.io/projected/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-kube-api-access-7mqds\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:19:28.165120 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.165100 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xtp\" (UniqueName: \"kubernetes.io/projected/2901d41f-f26c-4e6b-909f-1fc449e11ce8-kube-api-access-s8xtp\") pod \"network-check-source-6f78d59f99-t4nxw\" (UID: \"2901d41f-f26c-4e6b-909f-1fc449e11ce8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4nxw" Oct 31 03:19:28.166652 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.166636 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jgl\" (UniqueName: \"kubernetes.io/projected/4cc36af9-eaa8-40a3-b642-cc7aaf970ed6-kube-api-access-w8jgl\") pod \"kas-connection-checker-685895c7c9-pkgt6\" (UID: \"4cc36af9-eaa8-40a3-b642-cc7aaf970ed6\") " pod="kube-system/kas-connection-checker-685895c7c9-pkgt6" Oct 31 03:19:28.167314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.167300 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.195477 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195448 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-log-socket\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.195568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195484 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:28.195568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195508 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-kubernetes\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.195568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195531 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-systemd\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.195568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195554 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.195568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195561 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-log-socket\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195598 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-kubernetes\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195615 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195619 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-systemd\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195644 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6c807d9-708e-4527-8e56-11749a26ca61-hosts-file\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195606 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6c807d9-708e-4527-8e56-11749a26ca61-hosts-file\") pod \"node-resolver-kh444\" (UID: \"c6c807d9-708e-4527-8e56-11749a26ca61\") " pod="openshift-dns/node-resolver-kh444" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195673 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c58fbd8-7809-4718-8137-3439ac40c593-crio-socket\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195688 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-systemd-units\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.195623 4934 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195705 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-k8s-cni-cncf-io\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195725 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c58fbd8-7809-4718-8137-3439ac40c593-crio-socket\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.195761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195725 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-modprobe-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195766 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-systemd-units\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195752 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-netns\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195800 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-k8s-cni-cncf-io\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195811 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-netns\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.195818 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.695786107 +0000 UTC m=+2.014528738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195833 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-multus-certs\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195854 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-run\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195870 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-run-multus-certs\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195874 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-kubelet-config\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195880 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-modprobe-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195912 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-run\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195913 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-kubelet-config\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195938 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-kubelet\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195965 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195987 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-kubelet\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.195993 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196028 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-var-lib-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196051 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-sys\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196065 4934 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196067 4934 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196078 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-dbus\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196101 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-var-lib-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196103 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-slash\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196122 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696107604 +0000 UTC m=+2.014850249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : configmap "kube-rbac-proxy" not found Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196129 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-slash\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196140 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-sys\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196150 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696133677 +0000 UTC m=+2.014876287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:28.196158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196162 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3eefb160-a377-431f-be0a-c2bea6ecb352-dbus\") pod \"global-pull-secret-syncer-5ngpx\" (UID: \"3eefb160-a377-431f-be0a-c2bea6ecb352\") " pod="kube-system/global-pull-secret-syncer-5ngpx" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196183 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196205 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196223 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-registration-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196269 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196276 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196297 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-bin\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196312 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-cnibin\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196319 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196333 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196338 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196342 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-bin\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196367 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196388 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696374905 +0000 UTC m=+2.015117525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196320 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196402 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-registration-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196411 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-host\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196437 4934 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196442 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-cnibin\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196447 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196444 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6c13c0-234d-4042-a31c-ac2c374fd1b3-host\") pod \"node-ca-ntzrg\" (UID: \"ba6c13c0-234d-4042-a31c-ac2c374fd1b3\") " pod="openshift-image-registry/node-ca-ntzrg" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196476 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696469013 +0000 UTC m=+2.015211619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196507 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196537 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-netd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196542 4934 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196551 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-d\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196555 4934 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196555 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdd9600b-d5c9-4add-ae37-48aca3854510-host-slash\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196582 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdd9600b-d5c9-4add-ae37-48aca3854510-host-slash\") pod \"iptables-alerter-qgw8p\" (UID: \"bdd9600b-d5c9-4add-ae37-48aca3854510\") " pod="openshift-network-operator/iptables-alerter-qgw8p" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196595 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696580606 +0000 UTC m=+2.015323218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196602 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-cni-netd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196614 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696602352 +0000 UTC m=+2.015344959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196627 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196646 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196669 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196684 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-lib-modules\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196702 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-systemd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196711 4934 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196722 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-etc-kubernetes\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196734 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696722217 +0000 UTC m=+2.015464825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196753 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-etc-kubernetes\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196764 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-systemd\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196757 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696744986 +0000 UTC m=+2.015487626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : secret "metrics-daemon-secret" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196798 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-socket-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196803 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-lib-modules\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196817 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-etc-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196844 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196861 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196865 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-socket-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196797 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196877 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-kubelet\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196908 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196909 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-etc-openvswitch\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196923 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-kubelet\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196955 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196965 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.196972 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.196988 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.696980772 +0000 UTC m=+2.015723395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197005 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-device-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197023 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-host\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197036 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-cnibin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197007 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197052 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-ovn\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197072 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-run-ovn\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197080 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-node-log\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197099 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-host\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197115 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-device-dir\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197119 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-cnibin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197115 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-system-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197147 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197156 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-system-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197162 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-bin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197189 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-socket-dir-parent\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197214 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-socket-dir-parent\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197122 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-node-log\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197190 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-bin\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197250 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-cni-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197216 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-hostroot\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197286 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-hostroot\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197288 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-conf-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.197265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197313 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-netns\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197333 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-multus-conf-dir\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197339 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-conf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197360 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf4f6953-b84a-482d-b685-bd2addcbafde-host-run-netns\") pod \"ovnkube-node-gwqdn\" (UID: \"cf4f6953-b84a-482d-b685-bd2addcbafde\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197363 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197389 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-system-cni-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197429 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysconfig\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197440 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysctl-conf\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197472 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1687ced2-8f63-4900-9b49-275a8104e159-system-cni-dir\") pod \"multus-additional-cni-plugins-2r4ff\" (UID: \"1687ced2-8f63-4900-9b49-275a8104e159\") " pod="openshift-multus/multus-additional-cni-plugins-2r4ff" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197487 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-etc-sysconfig\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197499 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-var-lib-kubelet\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.197503 4934 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197520 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-multus\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197536 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b39f2c1f-e94d-4a91-aabd-08b7094aaa38-var-lib-kubelet\") pod \"tuned-hhs74\" (UID: \"b39f2c1f-e94d-4a91-aabd-08b7094aaa38\") " pod="openshift-cluster-node-tuning-operator/tuned-hhs74" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.197545 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.697530313 +0000 UTC m=+2.016272937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197554 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93cfa49d-0426-4d23-ad84-50f12272a871-host-var-lib-cni-multus\") pod \"multus-jw6nt\" (UID: \"93cfa49d-0426-4d23-ad84-50f12272a871\") " pod="openshift-multus/multus-jw6nt" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197562 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-sys-fs\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.199546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.197648 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cfceaec-d546-4c9e-ac02-e66eeeea477a-sys-fs\") pod \"aws-ebs-csi-driver-node-zncvb\" (UID: \"3cfceaec-d546-4c9e-ac02-e66eeeea477a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zncvb" Oct 31 03:19:28.229417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.229385 4934 scope.go:122] "RemoveContainer" containerID="94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4" Oct 31 03:19:28.702075 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.701978 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:28.702075 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702019 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.702075 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702040 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702122 4934 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702150 4934 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702185 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702171848 +0000 UTC m=+3.020914454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : configmap "kube-rbac-proxy" not found Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702185 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702210 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702194285 +0000 UTC m=+3.020936905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702273 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702272 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702283 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:28.702295 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702294 4934 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702307 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.70230012 +0000 UTC m=+3.021042727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702306 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702342 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702356 4934 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702368 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702378 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702372217 +0000 UTC m=+3.021114822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702391 4934 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702425 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702412728 +0000 UTC m=+3.021155335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702443 4934 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702455 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702449146 +0000 UTC m=+3.021191752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702466 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702459704 +0000 UTC m=+3.021202310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702485 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702475783 +0000 UTC m=+3.021218413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : secret "metrics-daemon-secret" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702444 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702506 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702515 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702529 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702548 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702540652 +0000 UTC m=+3.021283257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702570 4934 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.702582 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702590 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702584449 +0000 UTC m=+3.021327055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:28.702610 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702622 4934 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:28.703194 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.702650 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.702642511 +0000 UTC m=+3.021385117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:28.876616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.876584 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/2.log" Oct 31 03:19:28.877025 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.876991 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/1.log" Oct 31 03:19:28.877062 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.877021 4934 generic.go:356] "Generic (PLEG): container finished" podID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" containerID="de958499f72540a82ed9c5044c92cda8efdc4851c80e3236a973e4f4774ec357" exitCode=255 Oct 31 03:19:28.877118 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.877097 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" event={"ID":"9bdf0731-bef7-4c9c-a5c3-09eff5aef394","Type":"ContainerDied","Data":"de958499f72540a82ed9c5044c92cda8efdc4851c80e3236a973e4f4774ec357"} Oct 31 03:19:28.877166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.877156 4934 scope.go:122] "RemoveContainer" containerID="94e2c08f81ceffdbb40ce1e215a1487275308e44f62bcc250e6f06fc71fe7db4" Oct 31 03:19:28.877558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:28.877543 4934 scope.go:122] "RemoveContainer" containerID="de958499f72540a82ed9c5044c92cda8efdc4851c80e3236a973e4f4774ec357" Oct 31 03:19:28.877720 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:28.877698 4934 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:29.710220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710186 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710257 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710279 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710301 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710317 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710336 4934 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710351 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710380 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710402 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710381144 +0000 UTC m=+5.029123766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710401 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710415 4934 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710424 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710445 4934 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710459 4934 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710473 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710457598 +0000 UTC m=+5.029200227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710485 4934 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:29.710521 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710523 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710481418 +0000 UTC m=+5.029224027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : secret "metrics-daemon-secret" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710540 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls podName:cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710531207 +0000 UTC m=+5.029273813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l4jw7" (UID: "cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2") : secret "samples-operator-tls" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710555 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710547142 +0000 UTC m=+5.029289750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710572 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710617 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710644 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.71062758 +0000 UTC m=+5.029370195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710662 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710656182 +0000 UTC m=+5.029398788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710665 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710673 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710701 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710690479 +0000 UTC m=+5.029433091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710713 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710739 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710796 4934 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710803 4934 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710833 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710823113 +0000 UTC m=+5.029565738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:29.710846 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.710840217 +0000 UTC m=+5.029582823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:29.711201 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.710954 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c58fbd8-7809-4718-8137-3439ac40c593-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:29.884752 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:29.884721 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/2.log" Oct 31 03:19:30.057282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:30.057257 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kh444_c6c807d9-708e-4527-8e56-11749a26ca61/dns-node-resolver/0.log" Oct 31 03:19:30.261214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:30.261188 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:30.712498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:30.712461 4934 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:30.712820 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:30.712807 4934 scope.go:122] "RemoveContainer" containerID="de958499f72540a82ed9c5044c92cda8efdc4851c80e3236a973e4f4774ec357" Oct 31 03:19:30.713002 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:30.712984 4934 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:31.255470 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.255443 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ntzrg_ba6c13c0-234d-4042-a31c-ac2c374fd1b3/node-ca/0.log" Oct 31 03:19:31.516156 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.516065 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:31.516529 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.516511 4934 scope.go:122] "RemoveContainer" containerID="de958499f72540a82ed9c5044c92cda8efdc4851c80e3236a973e4f4774ec357" Oct 31 03:19:31.516747 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.516724 4934 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:31.655918 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.655886 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pvphc_2abc9860-5a1c-4ca0-a8ab-ee18d0a21b90/serve-healthcheck-canary/0.log" Oct 31 03:19:31.729023 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.728986 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729037 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729071 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729093 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729111 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729129 4934 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729139 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729168 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:31.729199 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729198 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729177316 +0000 UTC m=+9.047919945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729222 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729258 4934 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729257 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729262 4934 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729290 4934 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729259 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729306 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs podName:9ec0a926-6081-4a94-916a-2e27400b06a9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.72929076 +0000 UTC m=+9.048033380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs") pod "network-metrics-daemon-z2hw8" (UID: "9ec0a926-6081-4a94-916a-2e27400b06a9") : secret "metrics-daemon-secret" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729341 4934 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729365 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729398 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729379106 +0000 UTC m=+9.048121730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729426 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729416071 +0000 UTC m=+9.048158677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729455 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls podName:283b8014-5ffd-4671-a28d-2975376d9cf9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729435251 +0000 UTC m=+9.048177858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kkqzx" (UID: "283b8014-5ffd-4671-a28d-2975376d9cf9") : secret "cluster-monitoring-operator-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729459 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729473 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729476 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729469338 +0000 UTC m=+9.048211943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : secret "router-metrics-certs-default" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729492 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729486383 +0000 UTC m=+9.048228990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729515 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729501437 +0000 UTC m=+9.048244058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729535 4934 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:19:31.729587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.729568 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:31.730268 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729634 4934 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:31.730268 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:31.729687 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.729672063 +0000 UTC m=+9.048414684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:31.731796 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.731780 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l4jw7\" (UID: \"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:31.833488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:31.833441 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" Oct 31 03:19:32.200200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:32.200103 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7"] Oct 31 03:19:32.246873 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:32.246857 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:19:32.634452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:32.634424 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nv8wr_d0e050b4-e196-498c-bb05-3b1bc20f10ae/migrator/0.log" Oct 31 03:19:32.790792 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:32.790759 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h_a57d414f-4fb4-4b78-92c7-105e2ac19d21/kube-storage-version-migrator-operator/0.log" Oct 31 03:19:32.900553 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:32.900459 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" event={"ID":"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2","Type":"ContainerStarted","Data":"ce01aafd1a70aabd0930a9971cedd883aad12007a453eb66b3da3a0ec7c45227"} Oct 31 03:19:34.909192 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:34.909153 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" event={"ID":"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2","Type":"ContainerStarted","Data":"eeeb9f23adf7b5d777d91f651ff2a0614f01c1992f171cb8cd9224d5ab8a5ead"} Oct 31 03:19:34.909610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:34.909213 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l4jw7" event={"ID":"cb0ae5e6-cbd1-4f06-9e70-4bbf91c2bad2","Type":"ContainerStarted","Data":"f2c2bcf8ef8b261aa80874f0d07d198a3037038fea27441af3ffcd871d238ba4"} Oct 31 03:19:35.687449 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.687414 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:35.687651 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.687628 4934 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:35.767744 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.767705 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:35.767744 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.767748 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.767782 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.767802 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.767825 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.767866 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.767890 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f9f4c5f5-j4qrr: secret "image-registry-tls" not found Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.767928 4934 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.767956 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.767989 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls podName:6c58fbd8-7809-4718-8137-3439ac40c593 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:43.767970591 +0000 UTC m=+17.086713197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7s46" (UID: "6c58fbd8-7809-4718-8137-3439ac40c593") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:35.768017 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768016 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls podName:c6aa27c0-740f-4fa0-aadd-dbc207b19c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:43.768005825 +0000 UTC m=+17.086748431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls") pod "image-registry-79f9f4c5f5-j4qrr" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16") : secret "image-registry-tls" not found Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768057 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle podName:7c86d4eb-4dbf-4daf-95a3-62d73fc5210b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:43.76804052 +0000 UTC m=+17.086783140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle") pod "router-default-6d69c76ccf-dccpx" (UID: "7c86d4eb-4dbf-4daf-95a3-62d73fc5210b") : configmap references non-existent config key: service-ca.crt Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.768106 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.768181 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768290 4934 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768306 4934 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7548854-gr79z: secret "image-registry-tls" not found Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768351 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls podName:5df315ba-3f9f-4566-a1c0-d969f1430072 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:43.768332027 +0000 UTC m=+17.087074637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls") pod "image-registry-58d7548854-gr79z" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072") : secret "image-registry-tls" not found Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.768287 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:35.768471 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768466 4934 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:35.769052 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768487 4934 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 03:19:35.769052 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768504 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls podName:1c5fbb4c-6297-4655-8c67-ff336a28e948 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:43.768492403 +0000 UTC m=+17.087235011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls") pod "dns-default-4k5qr" (UID: "1c5fbb4c-6297-4655-8c67-ff336a28e948") : secret "dns-default-metrics-tls" not found Oct 31 03:19:35.769052 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:35.768589 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert podName:c3675be1-c31d-423f-83df-295df033f31a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:43.768517813 +0000 UTC m=+17.087260433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xbfpv" (UID: "c3675be1-c31d-423f-83df-295df033f31a") : secret "networking-console-plugin-cert" not found Oct 31 03:19:35.770745 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.770726 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec0a926-6081-4a94-916a-2e27400b06a9-metrics-certs\") pod \"network-metrics-daemon-z2hw8\" (UID: \"9ec0a926-6081-4a94-916a-2e27400b06a9\") " pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:35.770831 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.770815 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/283b8014-5ffd-4671-a28d-2975376d9cf9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kkqzx\" (UID: \"283b8014-5ffd-4671-a28d-2975376d9cf9\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:35.770929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:35.770907 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-metrics-certs\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:36.031723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:36.031638 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" Oct 31 03:19:36.037740 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:36.037713 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2hw8" Oct 31 03:19:36.262395 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:36.262361 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:36.406611 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:36.406576 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx"] Oct 31 03:19:36.444412 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:36.444381 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283b8014_5ffd_4671_a28d_2975376d9cf9.slice/crio-25c280d6bed864022e499aea784041b0dd7890a0335b0202f1976b8095f685cb WatchSource:0}: Error finding container 25c280d6bed864022e499aea784041b0dd7890a0335b0202f1976b8095f685cb: Status 404 returned error can't find the container with id 25c280d6bed864022e499aea784041b0dd7890a0335b0202f1976b8095f685cb Oct 31 03:19:36.452100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:36.452073 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2hw8"] Oct 31 03:19:36.455365 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:36.455339 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec0a926_6081_4a94_916a_2e27400b06a9.slice/crio-9d45d83325ec0223f32f742130dad122f1d65aae2cb5de5b9fc6caf1443aba0b WatchSource:0}: Error finding container 9d45d83325ec0223f32f742130dad122f1d65aae2cb5de5b9fc6caf1443aba0b: Status 404 returned error can't find the container with id 9d45d83325ec0223f32f742130dad122f1d65aae2cb5de5b9fc6caf1443aba0b Oct 31 03:19:36.916623 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:36.916585 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2hw8" event={"ID":"9ec0a926-6081-4a94-916a-2e27400b06a9","Type":"ContainerStarted","Data":"9d45d83325ec0223f32f742130dad122f1d65aae2cb5de5b9fc6caf1443aba0b"} Oct 31 03:19:36.917718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:36.917695 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" event={"ID":"283b8014-5ffd-4671-a28d-2975376d9cf9","Type":"ContainerStarted","Data":"25c280d6bed864022e499aea784041b0dd7890a0335b0202f1976b8095f685cb"} Oct 31 03:19:37.017761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:37.017730 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwqdn" Oct 31 03:19:37.312001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:37.311973 4934 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-238.us-west-1.compute.internal" event="NodeReady" Oct 31 03:19:37.925044 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:37.925002 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2hw8" event={"ID":"9ec0a926-6081-4a94-916a-2e27400b06a9","Type":"ContainerStarted","Data":"c3b0ec35ab83593abeb6345f784d2966ca18eb204283f05de44b7c6655e3be10"} Oct 31 03:19:38.929710 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:38.929667 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2hw8" event={"ID":"9ec0a926-6081-4a94-916a-2e27400b06a9","Type":"ContainerStarted","Data":"3633eaacd75dc97529ea96a8820216b02406f3e276e64b43bc321665f9d36d9c"} Oct 31 03:19:38.931314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:38.931285 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kkqzx" event={"ID":"283b8014-5ffd-4671-a28d-2975376d9cf9","Type":"ContainerStarted","Data":"70daf9f997e4e4f93e044527a6cccf32a8054bad79d722f064d76fcec5189f21"} Oct 31 03:19:41.818422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:41.815546 4934 scope.go:122] "RemoveContainer" containerID="de958499f72540a82ed9c5044c92cda8efdc4851c80e3236a973e4f4774ec357" Oct 31 03:19:41.942520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:41.942492 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:19:41.942902 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:41.942887 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/2.log" Oct 31 03:19:41.942946 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:41.942918 4934 generic.go:356] "Generic (PLEG): container finished" podID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" containerID="0fea001e7874a0991acab8a4fe90c6968bf5fafaad2e4be5d134bb197692e47f" exitCode=255 Oct 31 03:19:41.943016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:41.942994 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" event={"ID":"9bdf0731-bef7-4c9c-a5c3-09eff5aef394","Type":"ContainerDied","Data":"0fea001e7874a0991acab8a4fe90c6968bf5fafaad2e4be5d134bb197692e47f"} Oct 31 03:19:41.943090 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:41.943078 4934 scope.go:122] "RemoveContainer" containerID="de958499f72540a82ed9c5044c92cda8efdc4851c80e3236a973e4f4774ec357" Oct 31 03:19:41.943403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:41.943388 4934 scope.go:122] "RemoveContainer" containerID="0fea001e7874a0991acab8a4fe90c6968bf5fafaad2e4be5d134bb197692e47f" Oct 31 03:19:41.943618 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:41.943597 4934 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:42.948428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:42.948398 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:19:43.845374 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.845333 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:43.845589 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.845382 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:43.845589 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.845413 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:43.845589 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.845446 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:43.845827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.845607 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:43.845827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.845666 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:43.846251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.846209 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c86d4eb-4dbf-4daf-95a3-62d73fc5210b-service-ca-bundle\") pod \"router-default-6d69c76ccf-dccpx\" (UID: \"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b\") " pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:43.847906 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.847888 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c58fbd8-7809-4718-8137-3439ac40c593-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7s46\" (UID: \"6c58fbd8-7809-4718-8137-3439ac40c593\") " pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:43.848392 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.848371 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"image-registry-58d7548854-gr79z\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:43.848477 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.848432 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c5fbb4c-6297-4655-8c67-ff336a28e948-metrics-tls\") pod \"dns-default-4k5qr\" (UID: \"1c5fbb4c-6297-4655-8c67-ff336a28e948\") " pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:43.848518 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.848485 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"image-registry-79f9f4c5f5-j4qrr\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:43.848554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:43.848516 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3675be1-c31d-423f-83df-295df033f31a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xbfpv\" (UID: \"c3675be1-c31d-423f-83df-295df033f31a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:44.129310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.129207 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:44.134787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.134767 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:44.134853 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.134840 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" Oct 31 03:19:44.142428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.142403 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:44.143502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.143479 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z7s46" Oct 31 03:19:44.143682 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.143592 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:44.626921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.626891 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xbfpv"] Oct 31 03:19:44.630767 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:44.630733 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3675be1_c31d_423f_83df_295df033f31a.slice/crio-af0aef878de3c5b40c175555536e92638213bba5c187a60b2dd271aebf63d851 WatchSource:0}: Error finding container af0aef878de3c5b40c175555536e92638213bba5c187a60b2dd271aebf63d851: Status 404 returned error can't find the container with id af0aef878de3c5b40c175555536e92638213bba5c187a60b2dd271aebf63d851 Oct 31 03:19:44.633157 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:44.633137 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5fbb4c_6297_4655_8c67_ff336a28e948.slice/crio-1b2aa2ba77d180ae2e7fbf1f4ff4ee5a1d24e1b02fd829ffa34f5ebd71a760b2 WatchSource:0}: Error finding container 1b2aa2ba77d180ae2e7fbf1f4ff4ee5a1d24e1b02fd829ffa34f5ebd71a760b2: Status 404 returned error can't find the container with id 1b2aa2ba77d180ae2e7fbf1f4ff4ee5a1d24e1b02fd829ffa34f5ebd71a760b2 Oct 31 03:19:44.672793 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:44.672762 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aa27c0_740f_4fa0_aadd_dbc207b19c16.slice/crio-5440a33f2b32eb81b9b5e986cf2cffe491ac88cd4c10491b6b490ae967a3f475 WatchSource:0}: Error finding container 5440a33f2b32eb81b9b5e986cf2cffe491ac88cd4c10491b6b490ae967a3f475: Status 404 returned error can't find the container with id 5440a33f2b32eb81b9b5e986cf2cffe491ac88cd4c10491b6b490ae967a3f475 Oct 31 03:19:44.673273 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:44.673257 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df315ba_3f9f_4566_a1c0_d969f1430072.slice/crio-244d032ab0c867c1872fe5771b68020293c2fb017c9db9b582153369ccf6d05b WatchSource:0}: Error finding container 244d032ab0c867c1872fe5771b68020293c2fb017c9db9b582153369ccf6d05b: Status 404 returned error can't find the container with id 244d032ab0c867c1872fe5771b68020293c2fb017c9db9b582153369ccf6d05b Oct 31 03:19:44.676983 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.676960 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4k5qr"] Oct 31 03:19:44.680302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.680278 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d7548854-gr79z"] Oct 31 03:19:44.686763 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:44.686738 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c58fbd8_7809_4718_8137_3439ac40c593.slice/crio-77e947686c28214ce9ca982f9b05d5b3516560c50fa43d72fe9c0881147f7fb5 WatchSource:0}: Error finding container 77e947686c28214ce9ca982f9b05d5b3516560c50fa43d72fe9c0881147f7fb5: Status 404 returned error can't find the container with id 77e947686c28214ce9ca982f9b05d5b3516560c50fa43d72fe9c0881147f7fb5 Oct 31 03:19:44.687541 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:44.687523 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c86d4eb_4dbf_4daf_95a3_62d73fc5210b.slice/crio-a7be17f0e0966207be68c9acbdfdb3855678394e3af8db1fc62904117c07161b WatchSource:0}: Error finding container a7be17f0e0966207be68c9acbdfdb3855678394e3af8db1fc62904117c07161b: Status 404 returned error can't find the container with id a7be17f0e0966207be68c9acbdfdb3855678394e3af8db1fc62904117c07161b Oct 31 03:19:44.728445 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.728409 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f9f4c5f5-j4qrr"] Oct 31 03:19:44.728978 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.728950 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z7s46"] Oct 31 03:19:44.729537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.729483 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d69c76ccf-dccpx"] Oct 31 03:19:44.957693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.957586 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" event={"ID":"c3675be1-c31d-423f-83df-295df033f31a","Type":"ContainerStarted","Data":"af0aef878de3c5b40c175555536e92638213bba5c187a60b2dd271aebf63d851"} Oct 31 03:19:44.959042 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.959010 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4k5qr" event={"ID":"1c5fbb4c-6297-4655-8c67-ff336a28e948","Type":"ContainerStarted","Data":"1b2aa2ba77d180ae2e7fbf1f4ff4ee5a1d24e1b02fd829ffa34f5ebd71a760b2"} Oct 31 03:19:44.960760 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.960735 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" event={"ID":"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b","Type":"ContainerStarted","Data":"455163807fbd412c2bfe7dc0ad7f5e95ed1bdee78da2309b227419afbf234739"} Oct 31 03:19:44.960892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.960776 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" event={"ID":"7c86d4eb-4dbf-4daf-95a3-62d73fc5210b","Type":"ContainerStarted","Data":"a7be17f0e0966207be68c9acbdfdb3855678394e3af8db1fc62904117c07161b"} Oct 31 03:19:44.962482 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.962463 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7s46" event={"ID":"6c58fbd8-7809-4718-8137-3439ac40c593","Type":"ContainerStarted","Data":"82ea1224ef3e0a2b8815a24c7dde968d047c1c1437c10310e379f98c458f875d"} Oct 31 03:19:44.962575 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.962502 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7s46" event={"ID":"6c58fbd8-7809-4718-8137-3439ac40c593","Type":"ContainerStarted","Data":"77e947686c28214ce9ca982f9b05d5b3516560c50fa43d72fe9c0881147f7fb5"} Oct 31 03:19:44.964102 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.964082 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d7548854-gr79z" event={"ID":"5df315ba-3f9f-4566-a1c0-d969f1430072","Type":"ContainerStarted","Data":"39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd"} Oct 31 03:19:44.964185 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.964120 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d7548854-gr79z" event={"ID":"5df315ba-3f9f-4566-a1c0-d969f1430072","Type":"ContainerStarted","Data":"244d032ab0c867c1872fe5771b68020293c2fb017c9db9b582153369ccf6d05b"} Oct 31 03:19:44.964269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.964242 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:19:44.965800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.965780 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" event={"ID":"c6aa27c0-740f-4fa0-aadd-dbc207b19c16","Type":"ContainerStarted","Data":"6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd"} Oct 31 03:19:44.965871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.965810 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" event={"ID":"c6aa27c0-740f-4fa0-aadd-dbc207b19c16","Type":"ContainerStarted","Data":"5440a33f2b32eb81b9b5e986cf2cffe491ac88cd4c10491b6b490ae967a3f475"} Oct 31 03:19:44.965933 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:44.965916 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:19:45.055993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:45.055953 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f9f4c5f5-j4qrr"] Oct 31 03:19:45.135355 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:45.135313 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:45.138601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:45.138560 4934 patch_prober.go:29] interesting pod/router-default-6d69c76ccf-dccpx container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Oct 31 03:19:45.138601 ip-10-0-138-238 kubenswrapper[4934]: [-]has-synced failed: reason withheld Oct 31 03:19:45.138601 ip-10-0-138-238 kubenswrapper[4934]: [+]process-running ok Oct 31 03:19:45.138601 ip-10-0-138-238 kubenswrapper[4934]: healthz check failed Oct 31 03:19:45.138826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:45.138624 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" podUID="7c86d4eb-4dbf-4daf-95a3-62d73fc5210b" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 31 03:19:45.973391 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:45.973359 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7s46" event={"ID":"6c58fbd8-7809-4718-8137-3439ac40c593","Type":"ContainerStarted","Data":"d2fa686aa3668eaf6335d3b212908b63f256410079f5e38d11cccc150dd4a9ee"} Oct 31 03:19:46.136854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:46.136823 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:46.981624 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:46.981578 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4k5qr" event={"ID":"1c5fbb4c-6297-4655-8c67-ff336a28e948","Type":"ContainerStarted","Data":"7bcda1a48501a4eaec91c3128182b5929bf91c625bbf4fef9735ffd01d87d863"} Oct 31 03:19:46.987486 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:46.987454 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-xbfpv" event={"ID":"c3675be1-c31d-423f-83df-295df033f31a","Type":"ContainerStarted","Data":"bd488d574516bd3b22f4bbb4353b8fd49559a41bd2c01c688f98b400cd3f8145"} Oct 31 03:19:46.988562 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:46.988542 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:46.989937 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:46.989918 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d69c76ccf-dccpx" Oct 31 03:19:47.992545 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:47.992447 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7s46" event={"ID":"6c58fbd8-7809-4718-8137-3439ac40c593","Type":"ContainerStarted","Data":"5017ce636a06cc7a017e169d7db89e1ae3b9843f0b4b564eb99d217c6e8fc848"} Oct 31 03:19:47.994353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:47.994326 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4k5qr" event={"ID":"1c5fbb4c-6297-4655-8c67-ff336a28e948","Type":"ContainerStarted","Data":"a5fb420544b528c81361f77b6d7a13f59e3dcef6f9aad30ca3bb4ae5ef9a4f0b"} Oct 31 03:19:48.998163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:48.998126 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:49.162574 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:49.162540 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:19:50.713070 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:50.713028 4934 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:50.713588 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:50.713470 4934 scope.go:122] "RemoveContainer" containerID="0fea001e7874a0991acab8a4fe90c6968bf5fafaad2e4be5d134bb197692e47f" Oct 31 03:19:50.713707 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:50.713685 4934 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:51.517091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:51.517050 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:19:51.517485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:51.517471 4934 scope.go:122] "RemoveContainer" containerID="0fea001e7874a0991acab8a4fe90c6968bf5fafaad2e4be5d134bb197692e47f" Oct 31 03:19:51.517653 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:19:51.517635 4934 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b54xh_openshift-console-operator(9bdf0731-bef7-4c9c-a5c3-09eff5aef394)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" Oct 31 03:19:56.004487 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.004454 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4k5qr" Oct 31 03:19:56.265210 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.265137 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wdw6h" Oct 31 03:19:56.349158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.349124 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dgq2q"] Oct 31 03:19:56.394507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.394474 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.453817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.453775 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 03:19:56.453817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.453775 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 03:19:56.485110 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.485079 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 03:19:56.485311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.485178 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 03:19:56.485311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.485185 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m85g4\"" Oct 31 03:19:56.551827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.551742 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.551827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.551779 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4fc9\" (UniqueName: \"kubernetes.io/projected/f2cf121d-77e3-44a1-a62e-fd47b7504564-kube-api-access-k4fc9\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.552030 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.551877 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-tls\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.552030 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.551923 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-textfile\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.552030 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.551949 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-root\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.552030 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.551998 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.552163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.552103 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-sys\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.552163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.552135 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-wtmp\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.552163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.552154 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cf121d-77e3-44a1-a62e-fd47b7504564-metrics-client-ca\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653160 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653117 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-textfile\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653167 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-root\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653192 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653262 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-sys\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653297 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-wtmp\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653315 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-root\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653363 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-sys\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653701 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653408 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cf121d-77e3-44a1-a62e-fd47b7504564-metrics-client-ca\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653701 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653422 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-wtmp\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653701 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653484 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653701 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653510 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4fc9\" (UniqueName: \"kubernetes.io/projected/f2cf121d-77e3-44a1-a62e-fd47b7504564-kube-api-access-k4fc9\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653701 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653550 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-tls\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653701 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653559 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-textfile\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653896 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.653993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.653961 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cf121d-77e3-44a1-a62e-fd47b7504564-metrics-client-ca\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.656025 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.656001 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.656180 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.656159 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2cf121d-77e3-44a1-a62e-fd47b7504564-node-exporter-tls\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.714858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.714822 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4fc9\" (UniqueName: \"kubernetes.io/projected/f2cf121d-77e3-44a1-a62e-fd47b7504564-kube-api-access-k4fc9\") pod \"node-exporter-dgq2q\" (UID: \"f2cf121d-77e3-44a1-a62e-fd47b7504564\") " pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.746568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:56.746539 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dgq2q" Oct 31 03:19:56.785178 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:19:56.785145 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cf121d_77e3_44a1_a62e_fd47b7504564.slice/crio-3c81b4b448e6b073458d5a13d45f444478207b5e4cfd05b2a40f9896dc4e9463 WatchSource:0}: Error finding container 3c81b4b448e6b073458d5a13d45f444478207b5e4cfd05b2a40f9896dc4e9463: Status 404 returned error can't find the container with id 3c81b4b448e6b073458d5a13d45f444478207b5e4cfd05b2a40f9896dc4e9463 Oct 31 03:19:57.028306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:57.028268 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgq2q" event={"ID":"f2cf121d-77e3-44a1-a62e-fd47b7504564","Type":"ContainerStarted","Data":"3c81b4b448e6b073458d5a13d45f444478207b5e4cfd05b2a40f9896dc4e9463"} Oct 31 03:19:58.100140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:58.100104 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgq6v" Oct 31 03:19:59.039424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:59.039333 4934 generic.go:356] "Generic (PLEG): container finished" podID="f2cf121d-77e3-44a1-a62e-fd47b7504564" containerID="ed260a6c85b87d5cf8d756ad321f5e707ca88554c80090662f909f064edba8d1" exitCode=0 Oct 31 03:19:59.039699 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:19:59.039423 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgq2q" event={"ID":"f2cf121d-77e3-44a1-a62e-fd47b7504564","Type":"ContainerDied","Data":"ed260a6c85b87d5cf8d756ad321f5e707ca88554c80090662f909f064edba8d1"} Oct 31 03:20:00.045559 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:00.045523 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgq2q" event={"ID":"f2cf121d-77e3-44a1-a62e-fd47b7504564","Type":"ContainerStarted","Data":"795e12d55beba837e4dc8bf0ca190edce4cc6c2cc5ae5e224629eda8c036684e"} Oct 31 03:20:00.045946 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:00.045600 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgq2q" event={"ID":"f2cf121d-77e3-44a1-a62e-fd47b7504564","Type":"ContainerStarted","Data":"36619034bed3628aac4da254fad33ca4b70b5a3302ed781e8e650ffc315fc9bf"} Oct 31 03:20:00.165711 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:00.165650 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dgq2q" podStartSLOduration=2.272953032 podStartE2EDuration="4.165635734s" totalImagesPullingTime="1.814311821s" totalInitContainerRuntime="78.370881ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:00.16503172 +0000 UTC m=+33.483774349" watchObservedRunningTime="2026-10-31 03:20:00.165635734 +0000 UTC m=+33.484378363" Oct 31 03:20:04.819733 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:04.819707 4934 scope.go:122] "RemoveContainer" containerID="0fea001e7874a0991acab8a4fe90c6968bf5fafaad2e4be5d134bb197692e47f" Oct 31 03:20:05.066921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:05.066889 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:20:05.067099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:05.066974 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" event={"ID":"9bdf0731-bef7-4c9c-a5c3-09eff5aef394","Type":"ContainerStarted","Data":"a5548cba2fb6f8fcff19098a4970ca0da0d67661e0c04da86148889a331a147a"} Oct 31 03:20:05.067316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:05.067298 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:20:05.068454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:05.068424 4934 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-b54xh container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Oct 31 03:20:05.068584 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:05.068468 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" podUID="9bdf0731-bef7-4c9c-a5c3-09eff5aef394" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" Oct 31 03:20:05.978890 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:05.978862 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:20:06.075144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:06.075112 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b54xh" Oct 31 03:20:06.993980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:06.993952 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:20:07.573438 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:07.573406 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d7548854-gr79z"] Oct 31 03:20:12.012274 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.012214 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" podUID="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" containerName="registry" containerID="cri-o://6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd" gracePeriod=30 Oct 31 03:20:12.258469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.258442 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:20:12.278137 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278068 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278143 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278196 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278251 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278299 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278356 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278395 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278431 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") pod \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\" (UID: \"c6aa27c0-740f-4fa0-aadd-dbc207b19c16\") " Oct 31 03:20:12.278650 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278629 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:12.278730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.278705 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:12.281109 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.281079 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:12.281381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.281159 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "kube-api-access-58zth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:12.281381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.281189 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:12.281381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.281196 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:12.281777 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.281753 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:12.284928 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.284906 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted" pod "c6aa27c0-740f-4fa0-aadd-dbc207b19c16" (UID: "c6aa27c0-740f-4fa0-aadd-dbc207b19c16"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:12.379297 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379268 4934 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-certificates\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:12.379297 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379295 4934 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-installation-pull-secrets\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:12.379508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379306 4934 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-image-registry-private-configuration\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:12.379508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379316 4934 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-registry-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:12.379508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379327 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zth\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-kube-api-access-58zth\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:12.379508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379335 4934 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-trusted-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:12.379508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379343 4934 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-ca-trust-extracted\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:12.379508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:12.379352 4934 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6aa27c0-740f-4fa0-aadd-dbc207b19c16-bound-sa-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:13.100892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.100801 4934 generic.go:356] "Generic (PLEG): container finished" podID="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" containerID="6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd" exitCode=0 Oct 31 03:20:13.100892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.100842 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" event={"ID":"c6aa27c0-740f-4fa0-aadd-dbc207b19c16","Type":"ContainerDied","Data":"6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd"} Oct 31 03:20:13.100892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.100860 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" Oct 31 03:20:13.100892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.100895 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f9f4c5f5-j4qrr" event={"ID":"c6aa27c0-740f-4fa0-aadd-dbc207b19c16","Type":"ContainerDied","Data":"5440a33f2b32eb81b9b5e986cf2cffe491ac88cd4c10491b6b490ae967a3f475"} Oct 31 03:20:13.101480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.100921 4934 scope.go:122] "RemoveContainer" containerID="6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd" Oct 31 03:20:13.109389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.109368 4934 scope.go:122] "RemoveContainer" containerID="6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd" Oct 31 03:20:13.109697 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:20:13.109675 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd\": container with ID starting with 6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd not found: ID does not exist" containerID="6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd" Oct 31 03:20:13.109747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.109706 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd"} err="failed to get container status \"6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd\": rpc error: code = NotFound desc = could not find container \"6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd\": container with ID starting with 6785d5ceeffa9288cc4ea6ff7806e8948a4c658686cdf4aa4b6d51ac6ecf31dd not found: ID does not exist" Oct 31 03:20:13.318012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.317980 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f9f4c5f5-j4qrr"] Oct 31 03:20:13.368332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:13.368261 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79f9f4c5f5-j4qrr"] Oct 31 03:20:14.819948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:14.819914 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" path="/var/lib/kubelet/pods/c6aa27c0-740f-4fa0-aadd-dbc207b19c16/volumes" Oct 31 03:20:19.161773 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:20:19.161733 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:20:32.594056 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.593991 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58d7548854-gr79z" podUID="5df315ba-3f9f-4566-a1c0-d969f1430072" containerName="registry" containerID="cri-o://39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd" gracePeriod=30 Oct 31 03:20:32.829637 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.829611 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:20:32.944166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944084 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944131 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944175 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944205 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944271 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944311 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944352 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944392 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\") pod \"5df315ba-3f9f-4566-a1c0-d969f1430072\" (UID: \"5df315ba-3f9f-4566-a1c0-d969f1430072\") " Oct 31 03:20:32.944764 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.944734 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:32.945508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.945477 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:32.946862 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.946835 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:32.947006 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.946979 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "kube-api-access-n27js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:32.947080 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.947012 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:32.947156 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.947132 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:32.947200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.947176 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:32.956028 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:32.955999 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted" pod "5df315ba-3f9f-4566-a1c0-d969f1430072" (UID: "5df315ba-3f9f-4566-a1c0-d969f1430072"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:33.045976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.045940 4934 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-installation-pull-secrets\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.045976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.045971 4934 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-trusted-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.045976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.045981 4934 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.046221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.045991 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27js\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-kube-api-access-n27js\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.046221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.046000 4934 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df315ba-3f9f-4566-a1c0-d969f1430072-bound-sa-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.046221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.046008 4934 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5df315ba-3f9f-4566-a1c0-d969f1430072-ca-trust-extracted\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.046221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.046017 4934 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5df315ba-3f9f-4566-a1c0-d969f1430072-image-registry-private-configuration\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.046221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.046026 4934 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5df315ba-3f9f-4566-a1c0-d969f1430072-registry-certificates\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:33.174454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.174420 4934 generic.go:356] "Generic (PLEG): container finished" podID="5df315ba-3f9f-4566-a1c0-d969f1430072" containerID="39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd" exitCode=0 Oct 31 03:20:33.174644 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.174469 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7548854-gr79z" Oct 31 03:20:33.174644 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.174505 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d7548854-gr79z" event={"ID":"5df315ba-3f9f-4566-a1c0-d969f1430072","Type":"ContainerDied","Data":"39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd"} Oct 31 03:20:33.174644 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.174557 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d7548854-gr79z" event={"ID":"5df315ba-3f9f-4566-a1c0-d969f1430072","Type":"ContainerDied","Data":"244d032ab0c867c1872fe5771b68020293c2fb017c9db9b582153369ccf6d05b"} Oct 31 03:20:33.174644 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.174576 4934 scope.go:122] "RemoveContainer" containerID="39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd" Oct 31 03:20:33.183255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.183211 4934 scope.go:122] "RemoveContainer" containerID="39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd" Oct 31 03:20:33.183522 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:20:33.183506 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd\": container with ID starting with 39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd not found: ID does not exist" containerID="39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd" Oct 31 03:20:33.183563 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.183530 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd"} err="failed to get container status \"39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd\": rpc error: code = NotFound desc = could not find container \"39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd\": container with ID starting with 39e32464391f28d860f325feaaf4a30fdd24902dd9d427c983056bbdf481babd not found: ID does not exist" Oct 31 03:20:33.338696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.338669 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d7548854-gr79z"] Oct 31 03:20:33.344046 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:33.344022 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58d7548854-gr79z"] Oct 31 03:20:34.819574 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:34.819546 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df315ba-3f9f-4566-a1c0-d969f1430072" path="/var/lib/kubelet/pods/5df315ba-3f9f-4566-a1c0-d969f1430072/volumes" Oct 31 03:20:47.227786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:47.227748 4934 generic.go:356] "Generic (PLEG): container finished" podID="9f6df5b8-3430-46ee-8af1-13bb15bda816" containerID="4f81404d77f91f09c2fdfc818650a1411e025c066b241095194a1ef15f7b866c" exitCode=0 Oct 31 03:20:47.228202 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:47.227798 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zrtsh" event={"ID":"9f6df5b8-3430-46ee-8af1-13bb15bda816","Type":"ContainerDied","Data":"4f81404d77f91f09c2fdfc818650a1411e025c066b241095194a1ef15f7b866c"} Oct 31 03:20:47.228202 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:47.228117 4934 scope.go:122] "RemoveContainer" containerID="4f81404d77f91f09c2fdfc818650a1411e025c066b241095194a1ef15f7b866c" Oct 31 03:20:48.233572 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:48.233543 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zrtsh" event={"ID":"9f6df5b8-3430-46ee-8af1-13bb15bda816","Type":"ContainerStarted","Data":"4bbdd85cd0948c34de0f4b028cd3fb273dfd4746ef3391b17021ae1ea3c40e5c"} Oct 31 03:20:49.165985 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:20:49.165943 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:20:52.249784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:52.249754 4934 generic.go:356] "Generic (PLEG): container finished" podID="3aed8bd2-021f-4e33-96f0-e9c974fa0701" containerID="e1a278165148cb6279a2c2bcb41488f8ff213cd2221a452ee6a04fb5c6819599" exitCode=0 Oct 31 03:20:52.250162 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:52.249822 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" event={"ID":"3aed8bd2-021f-4e33-96f0-e9c974fa0701","Type":"ContainerDied","Data":"e1a278165148cb6279a2c2bcb41488f8ff213cd2221a452ee6a04fb5c6819599"} Oct 31 03:20:52.250162 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:52.250154 4934 scope.go:122] "RemoveContainer" containerID="e1a278165148cb6279a2c2bcb41488f8ff213cd2221a452ee6a04fb5c6819599" Oct 31 03:20:53.258153 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:20:53.258118 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpxww" event={"ID":"3aed8bd2-021f-4e33-96f0-e9c974fa0701","Type":"ContainerStarted","Data":"c029ad8d4c62e258919b58be5b99348908377ad87bbafe63144395cd1f17b014"} Oct 31 03:21:02.294496 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:21:02.294451 4934 generic.go:356] "Generic (PLEG): container finished" podID="a57d414f-4fb4-4b78-92c7-105e2ac19d21" containerID="942be70ca7fc3135e157e052f3a5ced4f43e9f237f3b9cef2d6acf547cd9e822" exitCode=0 Oct 31 03:21:02.294949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:21:02.294528 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" event={"ID":"a57d414f-4fb4-4b78-92c7-105e2ac19d21","Type":"ContainerDied","Data":"942be70ca7fc3135e157e052f3a5ced4f43e9f237f3b9cef2d6acf547cd9e822"} Oct 31 03:21:02.294949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:21:02.294825 4934 scope.go:122] "RemoveContainer" containerID="942be70ca7fc3135e157e052f3a5ced4f43e9f237f3b9cef2d6acf547cd9e822" Oct 31 03:21:03.300015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:21:03.299980 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gvn6h" event={"ID":"a57d414f-4fb4-4b78-92c7-105e2ac19d21","Type":"ContainerStarted","Data":"a172a959a5951af2302885a95e4939a8dd314a558155c7a58d309ac481f3ba1c"} Oct 31 03:21:19.161887 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:21:19.161846 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:21:49.161505 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:21:49.161413 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:22:19.162207 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:22:19.162171 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:22:49.162093 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:22:49.162054 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:23:19.162206 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:23:19.162166 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:23:19.766772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.766741 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb"] Oct 31 03:23:19.767041 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.767028 4934 cpu_manager.go:414] "containerMap: removing container" podUID="5df315ba-3f9f-4566-a1c0-d969f1430072" containerName="registry" Oct 31 03:23:19.767084 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.767043 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df315ba-3f9f-4566-a1c0-d969f1430072" containerName="registry" Oct 31 03:23:19.767084 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.767054 4934 cpu_manager.go:414] "containerMap: removing container" podUID="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" containerName="registry" Oct 31 03:23:19.767084 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.767059 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" containerName="registry" Oct 31 03:23:19.767174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.767122 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df315ba-3f9f-4566-a1c0-d969f1430072" containerName="registry" Oct 31 03:23:19.767174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.767131 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6aa27c0-740f-4fa0-aadd-dbc207b19c16" containerName="registry" Oct 31 03:23:19.770015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.769991 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.825048 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.825016 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:23:19.825315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.825297 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:23:19.825398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.825337 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:23:19.825457 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.825412 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:23:19.872921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.872891 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.873071 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.872985 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4chk\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-kube-api-access-w4chk\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.873071 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.873043 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11f4152-4e04-4eca-88d3-01cd36aff800-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.873175 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.873101 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.873285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.873270 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.873405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.873386 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.873548 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.873522 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.874952 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.874938 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-476rz\"" Oct 31 03:23:19.875873 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.875855 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb"] Oct 31 03:23:19.973896 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.973870 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.974083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.973908 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.974083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.973933 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.974083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.973952 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4chk\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-kube-api-access-w4chk\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.974083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.973982 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11f4152-4e04-4eca-88d3-01cd36aff800-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.974083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.974014 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.974083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.974038 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.974610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.974590 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.976375 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.976355 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11f4152-4e04-4eca-88d3-01cd36aff800-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.976502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.976484 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.976784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.976765 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:19.976887 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:19.976869 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:20.035507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:20.035438 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:20.035696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:20.035678 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4chk\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-kube-api-access-w4chk\") pod \"istiod-openshift-gateway-7876bfcb74-vschb\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:20.080876 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:20.080854 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:20.502421 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:20.502304 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb"] Oct 31 03:23:20.504913 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:23:20.504888 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11f4152_4e04_4eca_88d3_01cd36aff800.slice/crio-30a8f657e6f77b4448d6cd0c63207e02ccec8db3867adf5d6eb9297c80c7436b WatchSource:0}: Error finding container 30a8f657e6f77b4448d6cd0c63207e02ccec8db3867adf5d6eb9297c80c7436b: Status 404 returned error can't find the container with id 30a8f657e6f77b4448d6cd0c63207e02ccec8db3867adf5d6eb9297c80c7436b Oct 31 03:23:20.779941 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:20.779858 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" event={"ID":"a11f4152-4e04-4eca-88d3-01cd36aff800","Type":"ContainerStarted","Data":"30a8f657e6f77b4448d6cd0c63207e02ccec8db3867adf5d6eb9297c80c7436b"} Oct 31 03:23:24.330962 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:24.330932 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:23:24.331209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:24.330995 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:23:24.797188 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:24.797110 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" event={"ID":"a11f4152-4e04-4eca-88d3-01cd36aff800","Type":"ContainerStarted","Data":"20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65"} Oct 31 03:23:24.797424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:24.797255 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:25.016990 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:25.016936 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" podStartSLOduration=2.193410481 podStartE2EDuration="6.016923399s" totalImagesPullingTime="3.823512918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:24.966490397 +0000 UTC m=+238.285233025" watchObservedRunningTime="2026-10-31 03:23:25.016923399 +0000 UTC m=+238.335666025" Oct 31 03:23:25.802102 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:25.802064 4934 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-vschb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:23:25.802497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:25.802118 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" podUID="a11f4152-4e04-4eca-88d3-01cd36aff800" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:23:28.801627 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:28.801595 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:23:49.161711 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:23:49.161673 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:23:50.167456 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.167418 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd"] Oct 31 03:23:50.170712 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.170690 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.240849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.240820 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:23:50.273209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.273176 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:23:50.273209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.273204 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jccsh\"" Oct 31 03:23:50.273433 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.273192 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:23:50.273433 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.273270 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:23:50.273433 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.273355 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:23:50.275867 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.275842 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd"] Oct 31 03:23:50.320326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.320296 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcpg\" (UniqueName: \"kubernetes.io/projected/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-kube-api-access-wdcpg\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.320538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.320342 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.320538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.320377 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.320538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.320487 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.320538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.320533 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.320691 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.320561 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.421282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.421149 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.421282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.421206 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcpg\" (UniqueName: \"kubernetes.io/projected/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-kube-api-access-wdcpg\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.421282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.421280 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.421581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.421423 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.421581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.421476 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.421581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.421513 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.421727 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.421654 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.423899 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.423869 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.424039 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.424024 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.424080 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.424035 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.424080 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.424035 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.486877 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.486840 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcpg\" (UniqueName: \"kubernetes.io/projected/9d1d7e12-e19c-4eb2-a51e-7fb40df0573e-kube-api-access-wdcpg\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-xg4rd\" (UID: \"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:50.785897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:50.785821 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:51.394772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:51.394715 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd"] Oct 31 03:23:51.397498 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:23:51.397472 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1d7e12_e19c_4eb2_a51e_7fb40df0573e.slice/crio-ee1b58e66b2f7f32054898e4c48f2adc4488ed5a8260e18414f00bcc7bbcfe0e WatchSource:0}: Error finding container ee1b58e66b2f7f32054898e4c48f2adc4488ed5a8260e18414f00bcc7bbcfe0e: Status 404 returned error can't find the container with id ee1b58e66b2f7f32054898e4c48f2adc4488ed5a8260e18414f00bcc7bbcfe0e Oct 31 03:23:51.901140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:51.901107 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" event={"ID":"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e","Type":"ContainerStarted","Data":"ee1b58e66b2f7f32054898e4c48f2adc4488ed5a8260e18414f00bcc7bbcfe0e"} Oct 31 03:23:54.915970 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:54.915882 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" event={"ID":"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e","Type":"ContainerStarted","Data":"d126e529a3b30d4912c34108c3cb2e9e63ba5b3ec85c3a5bd3b85eb93c42901d"} Oct 31 03:23:55.921029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:55.920993 4934 generic.go:356] "Generic (PLEG): container finished" podID="9d1d7e12-e19c-4eb2-a51e-7fb40df0573e" containerID="d126e529a3b30d4912c34108c3cb2e9e63ba5b3ec85c3a5bd3b85eb93c42901d" exitCode=0 Oct 31 03:23:55.921427 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:55.921072 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" event={"ID":"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e","Type":"ContainerDied","Data":"d126e529a3b30d4912c34108c3cb2e9e63ba5b3ec85c3a5bd3b85eb93c42901d"} Oct 31 03:23:56.926373 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:56.926333 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" event={"ID":"9d1d7e12-e19c-4eb2-a51e-7fb40df0573e","Type":"ContainerStarted","Data":"bd2c8a8e8eb8628822eec25fc57b0c6fd47d75d33073d88dc32a8b3470acc047"} Oct 31 03:23:56.926768 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:56.926409 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:23:57.081468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:23:57.081419 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" podStartSLOduration=2.50969526 podStartE2EDuration="7.081406626s" totalImagesPullingTime="4.090926953s" totalInitContainerRuntime="480.784413ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:57.078805182 +0000 UTC m=+270.397547820" watchObservedRunningTime="2026-10-31 03:23:57.081406626 +0000 UTC m=+270.400149252" Oct 31 03:24:07.932909 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:07.932878 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-xg4rd" Oct 31 03:24:19.162977 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:24:19.162935 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:24:23.419031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.418996 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-764498d796-fmmn5"] Oct 31 03:24:23.422316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.422298 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.490420 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.490387 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f03c061-d61a-4f36-9a03-4bdd462df099-tmp\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.490591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.490431 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcv2\" (UniqueName: \"kubernetes.io/projected/8f03c061-d61a-4f36-9a03-4bdd462df099-kube-api-access-npcv2\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.490591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.490521 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f03c061-d61a-4f36-9a03-4bdd462df099-tls-certs\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.525091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.525061 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-764498d796-fmmn5"] Oct 31 03:24:23.547929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.547904 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 03:24:23.548036 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.547936 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:24:23.548036 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.547945 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cxr2k\"" Oct 31 03:24:23.591005 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.590977 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f03c061-d61a-4f36-9a03-4bdd462df099-tmp\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.591139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.591011 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcv2\" (UniqueName: \"kubernetes.io/projected/8f03c061-d61a-4f36-9a03-4bdd462df099-kube-api-access-npcv2\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.591139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.591056 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f03c061-d61a-4f36-9a03-4bdd462df099-tls-certs\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.593463 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.593436 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f03c061-d61a-4f36-9a03-4bdd462df099-tmp\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.593744 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.593722 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f03c061-d61a-4f36-9a03-4bdd462df099-tls-certs\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.652293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.652264 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcv2\" (UniqueName: \"kubernetes.io/projected/8f03c061-d61a-4f36-9a03-4bdd462df099-kube-api-access-npcv2\") pod \"kube-auth-proxy-764498d796-fmmn5\" (UID: \"8f03c061-d61a-4f36-9a03-4bdd462df099\") " pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:23.734095 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:23.734022 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" Oct 31 03:24:24.102958 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:24.102927 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-764498d796-fmmn5"] Oct 31 03:24:24.105663 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:24:24.105636 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f03c061_d61a_4f36_9a03_4bdd462df099.slice/crio-c494ffde1cf9171c0179c303f77e624a816819fa57050cec2372740159a86147 WatchSource:0}: Error finding container c494ffde1cf9171c0179c303f77e624a816819fa57050cec2372740159a86147: Status 404 returned error can't find the container with id c494ffde1cf9171c0179c303f77e624a816819fa57050cec2372740159a86147 Oct 31 03:24:25.032298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:25.032261 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" event={"ID":"8f03c061-d61a-4f36-9a03-4bdd462df099","Type":"ContainerStarted","Data":"c494ffde1cf9171c0179c303f77e624a816819fa57050cec2372740159a86147"} Oct 31 03:24:26.841276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:26.841245 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:24:26.841716 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:26.841366 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:24:26.985189 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:26.985164 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:24:28.047505 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:28.047469 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" event={"ID":"8f03c061-d61a-4f36-9a03-4bdd462df099","Type":"ContainerStarted","Data":"81ab6312f2dcc8763722e39f3fc390a543912dd372fe179b909c388cacb3c34c"} Oct 31 03:24:28.169873 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:28.169816 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-764498d796-fmmn5" podStartSLOduration=2.343988423 podStartE2EDuration="5.169804279s" totalImagesPullingTime="2.825815856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:28.16259211 +0000 UTC m=+301.481334739" watchObservedRunningTime="2026-10-31 03:24:28.169804279 +0000 UTC m=+301.488546900" Oct 31 03:24:31.622819 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.622784 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4"] Oct 31 03:24:31.628290 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.628273 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:31.677990 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.677947 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4"] Oct 31 03:24:31.726594 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.726553 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:24:31.726742 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.726641 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v67zc\"" Oct 31 03:24:31.726742 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.726722 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:24:31.762897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.762858 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2318925-8599-4d5c-9ba7-ca6626bfa997-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k9vh4\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:31.763007 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.762986 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p28m\" (UniqueName: \"kubernetes.io/projected/e2318925-8599-4d5c-9ba7-ca6626bfa997-kube-api-access-4p28m\") pod \"kuadrant-operator-controller-manager-678d87766f-k9vh4\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:31.863475 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.863439 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p28m\" (UniqueName: \"kubernetes.io/projected/e2318925-8599-4d5c-9ba7-ca6626bfa997-kube-api-access-4p28m\") pod \"kuadrant-operator-controller-manager-678d87766f-k9vh4\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:31.863654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.863486 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2318925-8599-4d5c-9ba7-ca6626bfa997-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k9vh4\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:31.863913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.863895 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2318925-8599-4d5c-9ba7-ca6626bfa997-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k9vh4\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:31.924784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.924712 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p28m\" (UniqueName: \"kubernetes.io/projected/e2318925-8599-4d5c-9ba7-ca6626bfa997-kube-api-access-4p28m\") pod \"kuadrant-operator-controller-manager-678d87766f-k9vh4\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:31.939802 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:31.939751 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:32.416742 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:32.416716 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4"] Oct 31 03:24:32.419565 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:24:32.419538 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2318925_8599_4d5c_9ba7_ca6626bfa997.slice/crio-119d095929088771000ba03fad93525b7e3291274fda4a33da3b7d3be85eb3b6 WatchSource:0}: Error finding container 119d095929088771000ba03fad93525b7e3291274fda4a33da3b7d3be85eb3b6: Status 404 returned error can't find the container with id 119d095929088771000ba03fad93525b7e3291274fda4a33da3b7d3be85eb3b6 Oct 31 03:24:32.422170 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:32.422155 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:24:33.069248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:33.069197 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" event={"ID":"e2318925-8599-4d5c-9ba7-ca6626bfa997","Type":"ContainerStarted","Data":"119d095929088771000ba03fad93525b7e3291274fda4a33da3b7d3be85eb3b6"} Oct 31 03:24:38.094311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:38.094269 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" event={"ID":"e2318925-8599-4d5c-9ba7-ca6626bfa997","Type":"ContainerStarted","Data":"f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1"} Oct 31 03:24:38.094679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:38.094372 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:38.314880 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:38.314826 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" podStartSLOduration=2.288473097 podStartE2EDuration="7.314809781s" totalImagesPullingTime="5.026336684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:38.263855623 +0000 UTC m=+311.582598270" watchObservedRunningTime="2026-10-31 03:24:38.314809781 +0000 UTC m=+311.633552409" Oct 31 03:24:49.101011 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:24:49.100978 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 03:24:49.162531 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:24:49.162493 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:25:19.162322 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:25:19.162287 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:25:49.162434 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:25:49.162384 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:25:54.725155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.725115 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q"] Oct 31 03:25:54.727307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.727291 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.792391 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.792363 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.792530 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.792400 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.792530 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.792425 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.792610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.792541 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.792648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.792608 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.792689 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.792653 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9z4c\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-kube-api-access-p9z4c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.792757 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.792741 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26a324-66d1-4a3a-a883-37d28fd726a1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.830937 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.830909 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q"] Oct 31 03:25:54.893534 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.893503 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9z4c\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-kube-api-access-p9z4c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.893715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.893548 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26a324-66d1-4a3a-a883-37d28fd726a1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.893715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.893581 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.893715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.893609 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.893715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.893634 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.893715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.893657 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.893715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.893705 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.894706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.894675 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.896467 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.896444 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26a324-66d1-4a3a-a883-37d28fd726a1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.896662 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.896645 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.896743 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.896724 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.896849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.896827 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.954218 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.954193 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:54.954564 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:54.954543 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9z4c\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-kube-api-access-p9z4c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trq9q\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:55.037181 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:55.037113 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:55.457591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:55.457553 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q"] Oct 31 03:25:55.460313 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:25:55.460287 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f26a324_66d1_4a3a_a883_37d28fd726a1.slice/crio-9eaa95cd1efee0defae4fc2cb711b504a4fe53e66b058e42653441d25331b7b5 WatchSource:0}: Error finding container 9eaa95cd1efee0defae4fc2cb711b504a4fe53e66b058e42653441d25331b7b5: Status 404 returned error can't find the container with id 9eaa95cd1efee0defae4fc2cb711b504a4fe53e66b058e42653441d25331b7b5 Oct 31 03:25:55.462537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:55.462508 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:25:55.462599 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:55.462569 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:25:56.400908 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:56.400872 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" event={"ID":"5f26a324-66d1-4a3a-a883-37d28fd726a1","Type":"ContainerStarted","Data":"18389bfc1402608f14b5c9bfebf7f0cf8d5fdb054526987d71817136a0f45564"} Oct 31 03:25:56.401411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:56.400921 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" event={"ID":"5f26a324-66d1-4a3a-a883-37d28fd726a1","Type":"ContainerStarted","Data":"9eaa95cd1efee0defae4fc2cb711b504a4fe53e66b058e42653441d25331b7b5"} Oct 31 03:25:56.401411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:56.401094 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:56.402679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:56.402653 4934 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-trq9q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:25:56.402793 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:56.402697 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" podUID="5f26a324-66d1-4a3a-a883-37d28fd726a1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:25:56.627617 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:56.627557 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" podStartSLOduration=2.627536457 podStartE2EDuration="2.627536457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:56.594873003 +0000 UTC m=+389.913615630" watchObservedRunningTime="2026-10-31 03:25:56.627536457 +0000 UTC m=+389.946279085" Oct 31 03:25:57.404954 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:57.404927 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:25:57.627122 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:57.627087 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb"] Oct 31 03:25:57.627343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:57.627319 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" podUID="a11f4152-4e04-4eca-88d3-01cd36aff800" containerName="discovery" containerID="cri-o://20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65" gracePeriod=30 Oct 31 03:25:57.867110 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:57.867086 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:25:58.018672 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.018584 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-kubeconfig\") pod \"a11f4152-4e04-4eca-88d3-01cd36aff800\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " Oct 31 03:25:58.018672 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.018626 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-kube-api-access-w4chk\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-kube-api-access-w4chk\") pod \"a11f4152-4e04-4eca-88d3-01cd36aff800\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " Oct 31 03:25:58.018672 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.018647 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-ca-configmap\") pod \"a11f4152-4e04-4eca-88d3-01cd36aff800\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " Oct 31 03:25:58.018940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.018688 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11f4152-4e04-4eca-88d3-01cd36aff800-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11f4152-4e04-4eca-88d3-01cd36aff800-local-certs\") pod \"a11f4152-4e04-4eca-88d3-01cd36aff800\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " Oct 31 03:25:58.018940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.018710 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-token\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-token\") pod \"a11f4152-4e04-4eca-88d3-01cd36aff800\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " Oct 31 03:25:58.018940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.018744 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-dns-cert\") pod \"a11f4152-4e04-4eca-88d3-01cd36aff800\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " Oct 31 03:25:58.018940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.018804 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-cacerts\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-cacerts\") pod \"a11f4152-4e04-4eca-88d3-01cd36aff800\" (UID: \"a11f4152-4e04-4eca-88d3-01cd36aff800\") " Oct 31 03:25:58.019218 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.019190 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-ca-configmap" pod "a11f4152-4e04-4eca-88d3-01cd36aff800" (UID: "a11f4152-4e04-4eca-88d3-01cd36aff800"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:58.021144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.021102 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11f4152-4e04-4eca-88d3-01cd36aff800-local-certs" pod "a11f4152-4e04-4eca-88d3-01cd36aff800" (UID: "a11f4152-4e04-4eca-88d3-01cd36aff800"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:58.021442 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.021417 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-kube-api-access-w4chk" pod "a11f4152-4e04-4eca-88d3-01cd36aff800" (UID: "a11f4152-4e04-4eca-88d3-01cd36aff800"). InnerVolumeSpecName "kube-api-access-w4chk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:58.021520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.021433 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-token" pod "a11f4152-4e04-4eca-88d3-01cd36aff800" (UID: "a11f4152-4e04-4eca-88d3-01cd36aff800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:58.021520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.021350 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-cacerts" pod "a11f4152-4e04-4eca-88d3-01cd36aff800" (UID: "a11f4152-4e04-4eca-88d3-01cd36aff800"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:58.021613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.021579 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-kubeconfig" pod "a11f4152-4e04-4eca-88d3-01cd36aff800" (UID: "a11f4152-4e04-4eca-88d3-01cd36aff800"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:58.021676 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.021627 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-dns-cert" pod "a11f4152-4e04-4eca-88d3-01cd36aff800" (UID: "a11f4152-4e04-4eca-88d3-01cd36aff800"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:58.120245 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.120189 4934 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-dns-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:25:58.120245 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.120219 4934 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-cacerts\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:25:58.120245 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.120240 4934 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-kubeconfig\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:25:58.120245 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.120249 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4chk\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-kube-api-access-w4chk\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:25:58.120503 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.120259 4934 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-csr-ca-configmap\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:25:58.120503 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.120268 4934 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11f4152-4e04-4eca-88d3-01cd36aff800-local-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:25:58.120503 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.120278 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11f4152-4e04-4eca-88d3-01cd36aff800-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:25:58.409023 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.408988 4934 generic.go:356] "Generic (PLEG): container finished" podID="a11f4152-4e04-4eca-88d3-01cd36aff800" containerID="20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65" exitCode=0 Oct 31 03:25:58.409472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.409037 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" Oct 31 03:25:58.409472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.409074 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" event={"ID":"a11f4152-4e04-4eca-88d3-01cd36aff800","Type":"ContainerDied","Data":"20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65"} Oct 31 03:25:58.409472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.409122 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb" event={"ID":"a11f4152-4e04-4eca-88d3-01cd36aff800","Type":"ContainerDied","Data":"30a8f657e6f77b4448d6cd0c63207e02ccec8db3867adf5d6eb9297c80c7436b"} Oct 31 03:25:58.409472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.409147 4934 scope.go:122] "RemoveContainer" containerID="20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65" Oct 31 03:25:58.417765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.417749 4934 scope.go:122] "RemoveContainer" containerID="20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65" Oct 31 03:25:58.418037 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:25:58.418018 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65\": container with ID starting with 20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65 not found: ID does not exist" containerID="20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65" Oct 31 03:25:58.418097 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.418050 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65"} err="failed to get container status \"20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65\": rpc error: code = NotFound desc = could not find container \"20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65\": container with ID starting with 20bd5d53b0e0e9eda68ec64965a54a7a22836cf52c8ca00cac8dc6b060f67a65 not found: ID does not exist" Oct 31 03:25:58.574486 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.574457 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb"] Oct 31 03:25:58.578343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.578318 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vschb"] Oct 31 03:25:58.819285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:58.819221 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11f4152-4e04-4eca-88d3-01cd36aff800" path="/var/lib/kubelet/pods/a11f4152-4e04-4eca-88d3-01cd36aff800/volumes" Oct 31 03:25:59.692871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.692835 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr"] Oct 31 03:25:59.693308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.693184 4934 cpu_manager.go:414] "containerMap: removing container" podUID="a11f4152-4e04-4eca-88d3-01cd36aff800" containerName="discovery" Oct 31 03:25:59.693308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.693197 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11f4152-4e04-4eca-88d3-01cd36aff800" containerName="discovery" Oct 31 03:25:59.693308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.693278 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11f4152-4e04-4eca-88d3-01cd36aff800" containerName="discovery" Oct 31 03:25:59.695615 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.695600 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:25:59.748107 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.748078 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 03:25:59.748282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.748180 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 03:25:59.748282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.748204 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 03:25:59.748396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.748302 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-g4x22\"" Oct 31 03:25:59.755945 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.755926 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr"] Oct 31 03:25:59.836636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.836606 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fe0ccd0-7848-4a34-a647-475c9ff012f9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l76tr\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:25:59.836796 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.836667 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978gr\" (UniqueName: \"kubernetes.io/projected/7fe0ccd0-7848-4a34-a647-475c9ff012f9-kube-api-access-978gr\") pod \"llmisvc-controller-manager-869d5dcc5d-l76tr\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:25:59.938029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.938002 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978gr\" (UniqueName: \"kubernetes.io/projected/7fe0ccd0-7848-4a34-a647-475c9ff012f9-kube-api-access-978gr\") pod \"llmisvc-controller-manager-869d5dcc5d-l76tr\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:25:59.938177 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.938060 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fe0ccd0-7848-4a34-a647-475c9ff012f9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l76tr\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:25:59.940664 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.940645 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fe0ccd0-7848-4a34-a647-475c9ff012f9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l76tr\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:25:59.999585 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:25:59.999521 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-978gr\" (UniqueName: \"kubernetes.io/projected/7fe0ccd0-7848-4a34-a647-475c9ff012f9-kube-api-access-978gr\") pod \"llmisvc-controller-manager-869d5dcc5d-l76tr\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:26:00.006417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:00.006396 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:26:00.377011 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:00.376982 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr"] Oct 31 03:26:00.379909 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:26:00.379881 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7fe0ccd0_7848_4a34_a647_475c9ff012f9.slice/crio-44f7e200b490eb2f86ebd467f67f25b31324c25b5a2a2e4e042100e0b6e1dd9a WatchSource:0}: Error finding container 44f7e200b490eb2f86ebd467f67f25b31324c25b5a2a2e4e042100e0b6e1dd9a: Status 404 returned error can't find the container with id 44f7e200b490eb2f86ebd467f67f25b31324c25b5a2a2e4e042100e0b6e1dd9a Oct 31 03:26:00.419128 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:00.419097 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" event={"ID":"7fe0ccd0-7848-4a34-a647-475c9ff012f9","Type":"ContainerStarted","Data":"44f7e200b490eb2f86ebd467f67f25b31324c25b5a2a2e4e042100e0b6e1dd9a"} Oct 31 03:26:03.433518 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.433486 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" event={"ID":"7fe0ccd0-7848-4a34-a647-475c9ff012f9","Type":"ContainerStarted","Data":"eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a"} Oct 31 03:26:03.433936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.433592 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:26:03.550828 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.550779 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" podStartSLOduration=2.5797929010000002 podStartE2EDuration="4.550764373s" totalImagesPullingTime="1.970971472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:03.549435829 +0000 UTC m=+396.868178456" watchObservedRunningTime="2026-10-31 03:26:03.550764373 +0000 UTC m=+396.869506999" Oct 31 03:26:03.854596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.854563 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2"] Oct 31 03:26:03.859309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.857956 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:03.960504 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.960463 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:26:03.960504 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.960466 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spbn9\"" Oct 31 03:26:03.960702 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.960516 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 03:26:03.960702 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.960519 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:26:03.960702 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.960546 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:26:03.961901 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.961880 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2"] Oct 31 03:26:03.972820 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.972800 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-metrics-certs\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:03.972928 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.972832 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtskd\" (UniqueName: \"kubernetes.io/projected/dfeebafe-bda1-4832-8c85-8564ad8febd7-kube-api-access-vtskd\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:03.972964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.972926 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-maas-api-tls\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:03.972996 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:03.972973 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeebafe-bda1-4832-8c85-8564ad8febd7-openshift-service-ca\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.073680 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.073646 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeebafe-bda1-4832-8c85-8564ad8febd7-openshift-service-ca\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.073861 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.073696 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-metrics-certs\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.073861 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.073719 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtskd\" (UniqueName: \"kubernetes.io/projected/dfeebafe-bda1-4832-8c85-8564ad8febd7-kube-api-access-vtskd\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.073861 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.073761 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-maas-api-tls\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.074318 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.074288 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeebafe-bda1-4832-8c85-8564ad8febd7-openshift-service-ca\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.076289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.076266 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-maas-api-tls\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.076371 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.076320 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-metrics-certs\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.134906 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.134843 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtskd\" (UniqueName: \"kubernetes.io/projected/dfeebafe-bda1-4832-8c85-8564ad8febd7-kube-api-access-vtskd\") pod \"maas-api-587c7c8bb4-kslb2\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.171181 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.171154 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:04.541864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:04.541725 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2"] Oct 31 03:26:04.544582 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:26:04.544545 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeebafe_bda1_4832_8c85_8564ad8febd7.slice/crio-c7c0fe9de154dcc5cabe6e7455cd604a532c4f49ed60eeeee0cdc002f3401434 WatchSource:0}: Error finding container c7c0fe9de154dcc5cabe6e7455cd604a532c4f49ed60eeeee0cdc002f3401434: Status 404 returned error can't find the container with id c7c0fe9de154dcc5cabe6e7455cd604a532c4f49ed60eeeee0cdc002f3401434 Oct 31 03:26:05.443136 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:05.443099 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" event={"ID":"dfeebafe-bda1-4832-8c85-8564ad8febd7","Type":"ContainerStarted","Data":"c7c0fe9de154dcc5cabe6e7455cd604a532c4f49ed60eeeee0cdc002f3401434"} Oct 31 03:26:07.454596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:07.454563 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" event={"ID":"dfeebafe-bda1-4832-8c85-8564ad8febd7","Type":"ContainerStarted","Data":"69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178"} Oct 31 03:26:07.454905 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:07.454656 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:07.615200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:07.615152 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" podStartSLOduration=1.7706071479999999 podStartE2EDuration="4.615134647s" totalImagesPullingTime="2.844527499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:07.614489616 +0000 UTC m=+400.933232244" watchObservedRunningTime="2026-10-31 03:26:07.615134647 +0000 UTC m=+400.933877267" Oct 31 03:26:13.464439 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:13.464413 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 03:26:19.161550 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:26:19.161509 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:26:34.440221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:26:34.440194 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:26:49.161963 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:26:49.161925 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:27:08.604060 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.604026 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b"] Oct 31 03:27:08.607446 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.607427 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:08.666567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.666531 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b"] Oct 31 03:27:08.696631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.696598 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a35468d-dc35-4c8b-9afe-e59d5c2b39be-cert\") pod \"llmisvc-controller-manager-6466c554d8-qbv2b\" (UID: \"3a35468d-dc35-4c8b-9afe-e59d5c2b39be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:08.696794 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.696651 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmkl\" (UniqueName: \"kubernetes.io/projected/3a35468d-dc35-4c8b-9afe-e59d5c2b39be-kube-api-access-rhmkl\") pod \"llmisvc-controller-manager-6466c554d8-qbv2b\" (UID: \"3a35468d-dc35-4c8b-9afe-e59d5c2b39be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:08.797306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.797264 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmkl\" (UniqueName: \"kubernetes.io/projected/3a35468d-dc35-4c8b-9afe-e59d5c2b39be-kube-api-access-rhmkl\") pod \"llmisvc-controller-manager-6466c554d8-qbv2b\" (UID: \"3a35468d-dc35-4c8b-9afe-e59d5c2b39be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:08.797479 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.797348 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a35468d-dc35-4c8b-9afe-e59d5c2b39be-cert\") pod \"llmisvc-controller-manager-6466c554d8-qbv2b\" (UID: \"3a35468d-dc35-4c8b-9afe-e59d5c2b39be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:08.799877 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.799859 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a35468d-dc35-4c8b-9afe-e59d5c2b39be-cert\") pod \"llmisvc-controller-manager-6466c554d8-qbv2b\" (UID: \"3a35468d-dc35-4c8b-9afe-e59d5c2b39be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:08.861568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.861502 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmkl\" (UniqueName: \"kubernetes.io/projected/3a35468d-dc35-4c8b-9afe-e59d5c2b39be-kube-api-access-rhmkl\") pod \"llmisvc-controller-manager-6466c554d8-qbv2b\" (UID: \"3a35468d-dc35-4c8b-9afe-e59d5c2b39be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:08.919372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:08.919344 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:09.289332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:09.289259 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b"] Oct 31 03:27:09.292914 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:27:09.292875 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a35468d_dc35_4c8b_9afe_e59d5c2b39be.slice/crio-2400973457e5ff06423282cfb474f93db9737cd9e820574b77fb405e92ced5e8 WatchSource:0}: Error finding container 2400973457e5ff06423282cfb474f93db9737cd9e820574b77fb405e92ced5e8: Status 404 returned error can't find the container with id 2400973457e5ff06423282cfb474f93db9737cd9e820574b77fb405e92ced5e8 Oct 31 03:27:09.707305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:09.707269 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" event={"ID":"3a35468d-dc35-4c8b-9afe-e59d5c2b39be","Type":"ContainerStarted","Data":"2400973457e5ff06423282cfb474f93db9737cd9e820574b77fb405e92ced5e8"} Oct 31 03:27:10.712148 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:10.712114 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" event={"ID":"3a35468d-dc35-4c8b-9afe-e59d5c2b39be","Type":"ContainerStarted","Data":"4a80be4c99632a5b80b6b93207ed9781375c5a8f461e447a95c8ea2acc2f1a63"} Oct 31 03:27:10.712564 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:10.712254 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:10.829950 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:10.829903 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" podStartSLOduration=2.003068847 podStartE2EDuration="2.829886506s" totalImagesPullingTime="826.817659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:10.829582234 +0000 UTC m=+464.148324859" watchObservedRunningTime="2026-10-31 03:27:10.829886506 +0000 UTC m=+464.148629130" Oct 31 03:27:19.162198 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:27:19.162163 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:27:41.719153 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:41.719073 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbv2b" Oct 31 03:27:41.886704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:41.886670 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr"] Oct 31 03:27:41.887172 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:41.887118 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" podUID="7fe0ccd0-7848-4a34-a647-475c9ff012f9" containerName="manager" containerID="cri-o://eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a" gracePeriod=30 Oct 31 03:27:42.133847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.133819 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:27:42.161211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.161180 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe0ccd0-7848-4a34-a647-475c9ff012f9-cert\" (UniqueName: \"kubernetes.io/secret/7fe0ccd0-7848-4a34-a647-475c9ff012f9-cert\") pod \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " Oct 31 03:27:42.161468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.161331 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0ccd0-7848-4a34-a647-475c9ff012f9-kube-api-access-978gr\" (UniqueName: \"kubernetes.io/projected/7fe0ccd0-7848-4a34-a647-475c9ff012f9-kube-api-access-978gr\") pod \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\" (UID: \"7fe0ccd0-7848-4a34-a647-475c9ff012f9\") " Oct 31 03:27:42.163367 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.163339 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe0ccd0-7848-4a34-a647-475c9ff012f9-cert" pod "7fe0ccd0-7848-4a34-a647-475c9ff012f9" (UID: "7fe0ccd0-7848-4a34-a647-475c9ff012f9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:27:42.163474 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.163451 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0ccd0-7848-4a34-a647-475c9ff012f9-kube-api-access-978gr" pod "7fe0ccd0-7848-4a34-a647-475c9ff012f9" (UID: "7fe0ccd0-7848-4a34-a647-475c9ff012f9"). InnerVolumeSpecName "kube-api-access-978gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:27:42.262365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.262282 4934 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fe0ccd0-7848-4a34-a647-475c9ff012f9-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:27:42.262365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.262312 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978gr\" (UniqueName: \"kubernetes.io/projected/7fe0ccd0-7848-4a34-a647-475c9ff012f9-kube-api-access-978gr\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:27:42.843239 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.843193 4934 generic.go:356] "Generic (PLEG): container finished" podID="7fe0ccd0-7848-4a34-a647-475c9ff012f9" containerID="eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a" exitCode=0 Oct 31 03:27:42.843652 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.843259 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" Oct 31 03:27:42.843652 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.843266 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" event={"ID":"7fe0ccd0-7848-4a34-a647-475c9ff012f9","Type":"ContainerDied","Data":"eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a"} Oct 31 03:27:42.843652 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.843318 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr" event={"ID":"7fe0ccd0-7848-4a34-a647-475c9ff012f9","Type":"ContainerDied","Data":"44f7e200b490eb2f86ebd467f67f25b31324c25b5a2a2e4e042100e0b6e1dd9a"} Oct 31 03:27:42.843652 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.843339 4934 scope.go:122] "RemoveContainer" containerID="eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a" Oct 31 03:27:42.851390 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.851374 4934 scope.go:122] "RemoveContainer" containerID="eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a" Oct 31 03:27:42.851640 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:27:42.851621 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a\": container with ID starting with eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a not found: ID does not exist" containerID="eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a" Oct 31 03:27:42.851718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:42.851647 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a"} err="failed to get container status \"eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a\": rpc error: code = NotFound desc = could not find container \"eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a\": container with ID starting with eabb267d2b27c4dcfe79ad4e64ffc5f852be1851cb82ed90b9ec5d74d9c0618a not found: ID does not exist" Oct 31 03:27:43.006554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:43.006522 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr"] Oct 31 03:27:43.056524 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:43.056485 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l76tr"] Oct 31 03:27:44.819805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:27:44.819772 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe0ccd0-7848-4a34-a647-475c9ff012f9" path="/var/lib/kubelet/pods/7fe0ccd0-7848-4a34-a647-475c9ff012f9/volumes" Oct 31 03:27:49.163250 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:27:49.163191 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:28:19.163411 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:28:19.163369 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:28:44.494449 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.494409 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp"] Oct 31 03:28:44.494901 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.494847 4934 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0ccd0-7848-4a34-a647-475c9ff012f9" containerName="manager" Oct 31 03:28:44.494901 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.494863 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0ccd0-7848-4a34-a647-475c9ff012f9" containerName="manager" Oct 31 03:28:44.495009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.494992 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0ccd0-7848-4a34-a647-475c9ff012f9" containerName="manager" Oct 31 03:28:44.497766 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.497745 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.550142 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.550108 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4td2m\"" Oct 31 03:28:44.559618 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.559591 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7q7\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-kube-api-access-dr7q7\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.559724 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.559637 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.559798 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.559720 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.559798 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.559781 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.559925 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.559894 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.560038 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.559946 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.560038 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.560011 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.560110 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.560073 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.560110 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.560097 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.601947 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.601917 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp"] Oct 31 03:28:44.660652 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.660625 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.660833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.660664 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.660833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.660688 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.660833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.660706 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.660833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.660724 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661104 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.660882 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661104 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661017 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661104 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661083 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7q7\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-kube-api-access-dr7q7\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661106 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661124 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661160 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661269 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661456 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661408 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.661687 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.661665 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.663304 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.663278 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.663729 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.663706 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.722599 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.722579 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.722871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.722851 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7q7\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-kube-api-access-dr7q7\") pod \"maas-default-gateway-openshift-default-fd4b99879-zphzp\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:44.810021 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:44.809997 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:45.233648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:45.233575 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp"] Oct 31 03:28:45.236662 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:28:45.236634 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9fe54c_889b_474c_94a1_20736e7dbe6d.slice/crio-23e556c78238b866430696705ef800e06cc9572f04c376da73ebbc52ff250350 WatchSource:0}: Error finding container 23e556c78238b866430696705ef800e06cc9572f04c376da73ebbc52ff250350: Status 404 returned error can't find the container with id 23e556c78238b866430696705ef800e06cc9572f04c376da73ebbc52ff250350 Oct 31 03:28:46.112672 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:46.112643 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" event={"ID":"9b9fe54c-889b-474c-94a1-20736e7dbe6d","Type":"ContainerStarted","Data":"23e556c78238b866430696705ef800e06cc9572f04c376da73ebbc52ff250350"} Oct 31 03:28:48.920296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:48.920263 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:28:48.920580 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:48.920329 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:28:48.920580 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:48.920357 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:28:49.128352 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:49.128311 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" event={"ID":"9b9fe54c-889b-474c-94a1-20736e7dbe6d","Type":"ContainerStarted","Data":"925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311"} Oct 31 03:28:49.163092 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:28:49.162985 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:28:49.299879 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:49.299774 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podStartSLOduration=1.618885675 podStartE2EDuration="5.299762222s" totalImagesPullingTime="3.680876547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:49.299071978 +0000 UTC m=+562.617814610" watchObservedRunningTime="2026-10-31 03:28:49.299762222 +0000 UTC m=+562.618504849" Oct 31 03:28:49.810459 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:49.810421 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:49.811645 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:49.811623 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-zphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:28:49.811764 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:49.811667 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 03:28:50.810500 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:50.810465 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-zphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:28:50.810981 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:50.810516 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 03:28:51.810846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:51.810811 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-zphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:28:51.811220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:51.810866 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 03:28:52.810723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:52.810685 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-zphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:28:52.810916 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:52.810745 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 03:28:53.811214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:53.811178 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-zphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:28:53.811682 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:53.811260 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 03:28:54.810553 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:54.810514 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-zphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:28:54.810756 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:54.810561 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 03:28:55.811158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:55.811123 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-zphzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:28:55.811686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:55.811173 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 03:28:56.874044 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:56.874006 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:56.874479 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:56.874354 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:56.875185 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:56.875162 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:28:59.626822 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.626788 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw"] Oct 31 03:28:59.630316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.630299 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682547 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682523 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682666 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682552 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682666 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682576 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ndj\" (UniqueName: \"kubernetes.io/projected/b9444d14-3d85-439f-a7ef-751d9fee021e-kube-api-access-d9ndj\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682666 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682641 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682682 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682737 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9444d14-3d85-439f-a7ef-751d9fee021e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682771 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682794 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.682891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.682859 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.783820 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.783792 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.783820 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.783823 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.783848 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ndj\" (UniqueName: \"kubernetes.io/projected/b9444d14-3d85-439f-a7ef-751d9fee021e-kube-api-access-d9ndj\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.783876 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.783913 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.783947 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9444d14-3d85-439f-a7ef-751d9fee021e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.783995 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.784031 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.784060 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784430 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.784407 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784531 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.784508 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.784679 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.784848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.784753 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.785248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.785205 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9444d14-3d85-439f-a7ef-751d9fee021e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.786916 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.786890 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.786916 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.786909 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.793185 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.793164 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw"] Oct 31 03:28:59.847493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.847471 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ndj\" (UniqueName: \"kubernetes.io/projected/b9444d14-3d85-439f-a7ef-751d9fee021e-kube-api-access-d9ndj\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.847591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.847542 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9444d14-3d85-439f-a7ef-751d9fee021e-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-q9mpw\" (UID: \"b9444d14-3d85-439f-a7ef-751d9fee021e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:28:59.942607 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:28:59.942533 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:29:00.366748 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:00.366726 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw"] Oct 31 03:29:00.369454 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:29:00.369427 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9444d14_3d85_439f_a7ef_751d9fee021e.slice/crio-a0c2b37bc436dadc120db7963909dc73a48469731362af23f7832b74b791e13b WatchSource:0}: Error finding container a0c2b37bc436dadc120db7963909dc73a48469731362af23f7832b74b791e13b: Status 404 returned error can't find the container with id a0c2b37bc436dadc120db7963909dc73a48469731362af23f7832b74b791e13b Oct 31 03:29:00.371409 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:00.371383 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:29:00.371484 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:00.371444 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:29:00.371484 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:00.371470 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:29:01.186073 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:01.186038 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" event={"ID":"b9444d14-3d85-439f-a7ef-751d9fee021e","Type":"ContainerStarted","Data":"e5b4d888022ec36bc751bcf22d8de8c684a1fb6cf86dc74abfedd0be683c1e61"} Oct 31 03:29:01.186073 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:01.186090 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" event={"ID":"b9444d14-3d85-439f-a7ef-751d9fee021e","Type":"ContainerStarted","Data":"a0c2b37bc436dadc120db7963909dc73a48469731362af23f7832b74b791e13b"} Oct 31 03:29:01.351563 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:01.351499 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podStartSLOduration=2.35148163 podStartE2EDuration="2.35148163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:01.350204384 +0000 UTC m=+574.668947014" watchObservedRunningTime="2026-10-31 03:29:01.35148163 +0000 UTC m=+574.670224257" Oct 31 03:29:01.943640 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:01.943608 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:29:01.944865 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:01.944843 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-q9mpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:29:01.944962 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:01.944884 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podUID="b9444d14-3d85-439f-a7ef-751d9fee021e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 03:29:02.943624 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:02.943592 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-q9mpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:29:02.943992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:02.943644 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podUID="b9444d14-3d85-439f-a7ef-751d9fee021e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 03:29:03.943859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:03.943823 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-q9mpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:29:03.944281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:03.943876 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podUID="b9444d14-3d85-439f-a7ef-751d9fee021e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 03:29:04.942984 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:04.942949 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-q9mpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:29:04.943176 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:04.943008 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podUID="b9444d14-3d85-439f-a7ef-751d9fee021e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 03:29:05.943391 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:05.943357 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-q9mpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:29:05.943856 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:05.943413 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podUID="b9444d14-3d85-439f-a7ef-751d9fee021e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 03:29:06.943069 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:06.943032 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-q9mpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:29:06.943244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:06.943100 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podUID="b9444d14-3d85-439f-a7ef-751d9fee021e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 03:29:08.311685 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:08.311651 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:29:08.312151 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:08.312020 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:29:08.313144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:08.313124 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" Oct 31 03:29:19.161798 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:29:19.161760 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:29:26.878409 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:26.878376 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:29:26.879027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:29:26.879005 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:29:49.162134 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:29:49.162095 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:30:19.161469 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:30:19.161435 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:30:22.088290 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.088257 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6"] Oct 31 03:30:22.091672 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.091653 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.144415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.144377 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-909365-openshift-default-dockercfg-rxcsh\"" Oct 31 03:30:22.164805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.164771 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.164957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.164883 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.164957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.164946 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.165045 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.164977 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.165045 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.165003 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.165119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.165055 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfh6w\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-kube-api-access-zfh6w\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.165175 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.165159 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.165217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.165205 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.165292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.165249 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.169438 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.169416 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6"] Oct 31 03:30:22.266649 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.266619 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.267465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.267437 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.268043 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.268025 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.269183 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.268935 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.269355 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.269056 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.269477 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.269462 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfh6w\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-kube-api-access-zfh6w\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.269603 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.269591 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.270026 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.270009 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.270877 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.270845 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.270992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.270914 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.270992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.269941 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.271109 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.269509 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.271184 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.270779 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.271980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.271249 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.272137 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.271522 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.273810 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.273786 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.329399 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.329370 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.330485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.330462 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfh6w\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-kube-api-access-zfh6w\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.404321 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.404223 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:22.832500 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.832478 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6"] Oct 31 03:30:22.835131 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:30:22.835100 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5682ca43_69b2_4c07_b89e_4b5f062f6c67.slice/crio-645e23bbddf3b74b7983cc9613fd7c8d06920445e53110015e9cfc9669e9db8b WatchSource:0}: Error finding container 645e23bbddf3b74b7983cc9613fd7c8d06920445e53110015e9cfc9669e9db8b: Status 404 returned error can't find the container with id 645e23bbddf3b74b7983cc9613fd7c8d06920445e53110015e9cfc9669e9db8b Oct 31 03:30:22.837182 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.837156 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:30:22.837286 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.837216 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:30:22.837286 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:22.837272 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:30:23.527713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:23.527676 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" event={"ID":"5682ca43-69b2-4c07-b89e-4b5f062f6c67","Type":"ContainerStarted","Data":"b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4"} Oct 31 03:30:23.528157 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:23.527732 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" event={"ID":"5682ca43-69b2-4c07-b89e-4b5f062f6c67","Type":"ContainerStarted","Data":"645e23bbddf3b74b7983cc9613fd7c8d06920445e53110015e9cfc9669e9db8b"} Oct 31 03:30:23.704921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:23.704874 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" podStartSLOduration=1.704862634 podStartE2EDuration="1.704862634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:23.701393271 +0000 UTC m=+657.020135898" watchObservedRunningTime="2026-10-31 03:30:23.704862634 +0000 UTC m=+657.023605308" Oct 31 03:30:24.404559 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:24.404524 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:24.409514 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:24.409489 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:24.532214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:24.532185 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:24.533116 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:24.533099 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:30:46.556607 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.556529 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x"] Oct 31 03:30:46.559952 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.559930 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.612318 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.612294 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b98893-serving-cert\"" Oct 31 03:30:46.663428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.663402 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x"] Oct 31 03:30:46.685964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.685939 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44742f8-e136-48da-95c6-8cd88b73e76f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.686081 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.686009 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.686128 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.686083 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjdh\" (UniqueName: \"kubernetes.io/projected/c44742f8-e136-48da-95c6-8cd88b73e76f-kube-api-access-8rjdh\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.686128 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.686109 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-metrics-certs\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.786564 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.786525 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44742f8-e136-48da-95c6-8cd88b73e76f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.786717 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.786611 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.786717 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.786640 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjdh\" (UniqueName: \"kubernetes.io/projected/c44742f8-e136-48da-95c6-8cd88b73e76f-kube-api-access-8rjdh\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.786717 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.786655 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-metrics-certs\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.786819 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:30:46.786762 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-serving-cert: secret "maas-api-e2e-worker-w3-b98893-serving-cert" not found Oct 31 03:30:46.786859 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:30:46.786838 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls podName:c44742f8-e136-48da-95c6-8cd88b73e76f nodeName:}" failed. No retries permitted until 2026-10-31 03:30:47.286818401 +0000 UTC m=+680.605561022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls") pod "maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" (UID: "c44742f8-e136-48da-95c6-8cd88b73e76f") : secret "maas-api-e2e-worker-w3-b98893-serving-cert" not found Oct 31 03:30:46.787292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.787270 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44742f8-e136-48da-95c6-8cd88b73e76f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.789162 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.789136 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-metrics-certs\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:46.853126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:46.853049 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjdh\" (UniqueName: \"kubernetes.io/projected/c44742f8-e136-48da-95c6-8cd88b73e76f-kube-api-access-8rjdh\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:47.292465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:47.292379 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:47.295295 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:47.295266 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b98893-666759d68c-95m9x\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:47.471799 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:47.471768 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:47.850974 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:47.850945 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x"] Oct 31 03:30:47.853937 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:30:47.853906 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44742f8_e136_48da_95c6_8cd88b73e76f.slice/crio-e85ebb883f7eca8e2ab964914bbce5b16a3cd9191e1bf0dd0a755a4ff23a7854 WatchSource:0}: Error finding container e85ebb883f7eca8e2ab964914bbce5b16a3cd9191e1bf0dd0a755a4ff23a7854: Status 404 returned error can't find the container with id e85ebb883f7eca8e2ab964914bbce5b16a3cd9191e1bf0dd0a755a4ff23a7854 Oct 31 03:30:47.855140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:47.855123 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:30:48.638170 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:48.638145 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" event={"ID":"c44742f8-e136-48da-95c6-8cd88b73e76f","Type":"ContainerStarted","Data":"e85ebb883f7eca8e2ab964914bbce5b16a3cd9191e1bf0dd0a755a4ff23a7854"} Oct 31 03:30:49.163184 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:30:49.163134 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:30:49.644570 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:49.644539 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" event={"ID":"c44742f8-e136-48da-95c6-8cd88b73e76f","Type":"ContainerStarted","Data":"0eef1d0251578d33ed1ef9d39d0d13e65ade46192efd9b217b5bdcf3249e89e9"} Oct 31 03:30:49.644735 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:49.644642 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:49.762371 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:49.762322 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" podStartSLOduration=3.001054062 podStartE2EDuration="3.762309399s" totalImagesPullingTime="761.255337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:49.761507345 +0000 UTC m=+683.080249990" watchObservedRunningTime="2026-10-31 03:30:49.762309399 +0000 UTC m=+683.081052042" Oct 31 03:30:55.654328 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:55.654301 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:30:56.550244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.550195 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc"] Oct 31 03:30:56.553533 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.553512 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.587120 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.587085 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febda22b-0402-4a68-a578-4202936a45b7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.587307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.587208 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbl2\" (UniqueName: \"kubernetes.io/projected/febda22b-0402-4a68-a578-4202936a45b7-kube-api-access-fmbl2\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.587307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.587270 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.587307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.587299 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-metrics-certs\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.605747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.605712 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-909365-serving-cert\"" Oct 31 03:30:56.657864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.657824 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc"] Oct 31 03:30:56.688065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.688032 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febda22b-0402-4a68-a578-4202936a45b7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.688264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.688121 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbl2\" (UniqueName: \"kubernetes.io/projected/febda22b-0402-4a68-a578-4202936a45b7-kube-api-access-fmbl2\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.688264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.688152 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.688264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.688180 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-metrics-certs\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.688751 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:30:56.688716 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-serving-cert: secret "maas-api-e2e-worker-w0-909365-serving-cert" not found Oct 31 03:30:56.688968 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:30:56.688940 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls podName:febda22b-0402-4a68-a578-4202936a45b7 nodeName:}" failed. No retries permitted until 2026-10-31 03:30:57.188918299 +0000 UTC m=+690.507660906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls") pod "maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" (UID: "febda22b-0402-4a68-a578-4202936a45b7") : secret "maas-api-e2e-worker-w0-909365-serving-cert" not found Oct 31 03:30:56.688968 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.688936 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febda22b-0402-4a68-a578-4202936a45b7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.691847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.691823 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-metrics-certs\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:56.750191 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:56.750165 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbl2\" (UniqueName: \"kubernetes.io/projected/febda22b-0402-4a68-a578-4202936a45b7-kube-api-access-fmbl2\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:57.195525 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:57.195478 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:57.198156 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:57.198134 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-909365-55fc57869-ndjlc\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:57.475987 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:57.475911 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:30:57.849776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:57.849743 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc"] Oct 31 03:30:57.855142 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:30:57.855116 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebda22b_0402_4a68_a578_4202936a45b7.slice/crio-1c1b48d0c9a80c1d3b637ae1d65e2db667b08adfe030f34f1f50b38d8d52890d WatchSource:0}: Error finding container 1c1b48d0c9a80c1d3b637ae1d65e2db667b08adfe030f34f1f50b38d8d52890d: Status 404 returned error can't find the container with id 1c1b48d0c9a80c1d3b637ae1d65e2db667b08adfe030f34f1f50b38d8d52890d Oct 31 03:30:58.686854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:58.686828 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" event={"ID":"febda22b-0402-4a68-a578-4202936a45b7","Type":"ContainerStarted","Data":"1c1b48d0c9a80c1d3b637ae1d65e2db667b08adfe030f34f1f50b38d8d52890d"} Oct 31 03:30:59.693056 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:59.693025 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" event={"ID":"febda22b-0402-4a68-a578-4202936a45b7","Type":"ContainerStarted","Data":"508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41"} Oct 31 03:30:59.693451 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:30:59.693085 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:31:05.709009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:05.708980 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 03:31:05.836869 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:05.836809 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" podStartSLOduration=9.030613533 podStartE2EDuration="9.83679578s" totalImagesPullingTime="806.182247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:59.811860314 +0000 UTC m=+693.130602943" watchObservedRunningTime="2026-10-31 03:31:05.83679578 +0000 UTC m=+699.155538413" Oct 31 03:31:07.605646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.605612 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5"] Oct 31 03:31:07.609497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.609474 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.672250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.671529 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-14cb5a-serving-cert\"" Oct 31 03:31:07.721792 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.721761 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc22v\" (UniqueName: \"kubernetes.io/projected/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-kube-api-access-vc22v\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.721956 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.721828 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-metrics-certs\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.721956 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.721897 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.721956 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.721941 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.722964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.722941 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5"] Oct 31 03:31:07.822898 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.822858 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc22v\" (UniqueName: \"kubernetes.io/projected/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-kube-api-access-vc22v\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.823087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.822932 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-metrics-certs\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.823087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.822959 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.823087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.822991 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.823286 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:31:07.823083 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-serving-cert: secret "maas-api-e2e-worker-w5-14cb5a-serving-cert" not found Oct 31 03:31:07.823286 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:31:07.823149 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls podName:f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2 nodeName:}" failed. No retries permitted until 2026-10-31 03:31:08.323132408 +0000 UTC m=+701.641875017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls") pod "maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" (UID: "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2") : secret "maas-api-e2e-worker-w5-14cb5a-serving-cert" not found Oct 31 03:31:07.823730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.823711 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.825562 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.825545 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-metrics-certs\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:07.894606 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:07.894517 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc22v\" (UniqueName: \"kubernetes.io/projected/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-kube-api-access-vc22v\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:08.328531 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:08.328494 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:08.330936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:08.330913 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:08.521544 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:08.521511 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:08.919673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:08.919597 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5"] Oct 31 03:31:08.922983 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:31:08.922950 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf731aec1_c19b_4e31_a5dc_0b7bbc8d55f2.slice/crio-ac95a588a42655e6c1a7f3f2a89224558c3f4a73e9fc2b95008c618be141adde WatchSource:0}: Error finding container ac95a588a42655e6c1a7f3f2a89224558c3f4a73e9fc2b95008c618be141adde: Status 404 returned error can't find the container with id ac95a588a42655e6c1a7f3f2a89224558c3f4a73e9fc2b95008c618be141adde Oct 31 03:31:09.738063 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:09.738024 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" event={"ID":"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2","Type":"ContainerStarted","Data":"ac95a588a42655e6c1a7f3f2a89224558c3f4a73e9fc2b95008c618be141adde"} Oct 31 03:31:10.745447 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:10.745405 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" event={"ID":"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2","Type":"ContainerStarted","Data":"6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee"} Oct 31 03:31:10.745879 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:10.745644 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:10.863981 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:10.863913 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" podStartSLOduration=2.9979444920000002 podStartE2EDuration="3.863893897s" totalImagesPullingTime="865.949405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:10.86331317 +0000 UTC m=+704.182055799" watchObservedRunningTime="2026-10-31 03:31:10.863893897 +0000 UTC m=+704.182636527" Oct 31 03:31:13.760788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:13.760753 4934 generic.go:356] "Generic (PLEG): container finished" podID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerID="b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4" exitCode=0 Oct 31 03:31:13.761248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:13.760804 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" event={"ID":"5682ca43-69b2-4c07-b89e-4b5f062f6c67","Type":"ContainerDied","Data":"b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4"} Oct 31 03:31:13.761248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:13.761163 4934 scope.go:122] "RemoveContainer" containerID="b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4" Oct 31 03:31:13.762407 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:13.762378 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:31:13.762505 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:13.762443 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:31:13.762505 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:13.762472 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:31:14.784849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:14.784814 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" event={"ID":"5682ca43-69b2-4c07-b89e-4b5f062f6c67","Type":"ContainerStarted","Data":"52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195"} Oct 31 03:31:15.405342 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:15.405304 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:31:15.405550 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:15.405508 4934 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:15.405630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:15.405566 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 03:31:16.404806 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:16.404768 4934 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:16.405193 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:16.404831 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 03:31:16.763314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:16.762715 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 03:31:17.405027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:17.404994 4934 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:17.405433 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:17.405044 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 03:31:18.157252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.155557 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n"] Oct 31 03:31:18.176907 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.176882 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.221711 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.221677 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbjc\" (UniqueName: \"kubernetes.io/projected/9da424b7-886e-4ace-985b-d6f5f4b631d0-kube-api-access-9xbjc\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.221838 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.221739 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.221838 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.221774 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da424b7-886e-4ace-985b-d6f5f4b631d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.221838 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.221826 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-metrics-certs\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.222043 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.222022 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n"] Oct 31 03:31:18.285936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.285815 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-dec4a0-serving-cert\"" Oct 31 03:31:18.322617 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.322590 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbjc\" (UniqueName: \"kubernetes.io/projected/9da424b7-886e-4ace-985b-d6f5f4b631d0-kube-api-access-9xbjc\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.322808 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.322636 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.322808 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.322662 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da424b7-886e-4ace-985b-d6f5f4b631d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.322808 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.322702 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-metrics-certs\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.322808 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:31:18.322770 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-serving-cert: secret "maas-api-e2e-worker-w2-dec4a0-serving-cert" not found Oct 31 03:31:18.323011 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:31:18.322852 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls podName:9da424b7-886e-4ace-985b-d6f5f4b631d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:31:18.822828972 +0000 UTC m=+712.141571595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls") pod "maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" (UID: "9da424b7-886e-4ace-985b-d6f5f4b631d0") : secret "maas-api-e2e-worker-w2-dec4a0-serving-cert" not found Oct 31 03:31:18.323478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.323454 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da424b7-886e-4ace-985b-d6f5f4b631d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.325171 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.325152 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-metrics-certs\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.384638 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.384611 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbjc\" (UniqueName: \"kubernetes.io/projected/9da424b7-886e-4ace-985b-d6f5f4b631d0-kube-api-access-9xbjc\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.407105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.406684 4934 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:18.407105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.406745 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 03:31:18.828283 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.828248 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:18.830858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:18.830789 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:19.095028 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:19.094951 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:19.195276 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:31:19.195213 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:31:19.404993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:19.404907 4934 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:19.404993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:19.404963 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 03:31:19.468562 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:19.468536 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n"] Oct 31 03:31:19.471010 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:31:19.470983 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da424b7_886e_4ace_985b_d6f5f4b631d0.slice/crio-80ceeb94cbdd368dfe8ae80373d17f8253ace97f941ee524739d06ba2927fe61 WatchSource:0}: Error finding container 80ceeb94cbdd368dfe8ae80373d17f8253ace97f941ee524739d06ba2927fe61: Status 404 returned error can't find the container with id 80ceeb94cbdd368dfe8ae80373d17f8253ace97f941ee524739d06ba2927fe61 Oct 31 03:31:19.809700 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:19.809669 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" event={"ID":"9da424b7-886e-4ace-985b-d6f5f4b631d0","Type":"ContainerStarted","Data":"80ceeb94cbdd368dfe8ae80373d17f8253ace97f941ee524739d06ba2927fe61"} Oct 31 03:31:20.409806 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:20.409772 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:31:20.410167 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:20.410150 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:31:20.410848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:20.410830 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 03:31:20.819336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:20.819300 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" event={"ID":"9da424b7-886e-4ace-985b-d6f5f4b631d0","Type":"ContainerStarted","Data":"723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f"} Oct 31 03:31:20.942091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:20.942041 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" podStartSLOduration=2.100279127 podStartE2EDuration="2.942026813s" totalImagesPullingTime="841.747686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:20.940466164 +0000 UTC m=+714.259208803" watchObservedRunningTime="2026-10-31 03:31:20.942026813 +0000 UTC m=+714.260769438" Oct 31 03:31:21.819612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:21.819584 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:27.830007 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:31:27.829973 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 03:31:49.163206 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:31:49.163174 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:32:07.339335 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.339257 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g"] Oct 31 03:32:07.343930 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.343911 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.384133 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.384105 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-model-cache\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.384289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.384159 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-tmp-dir\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.384289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.384268 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzdb\" (UniqueName: \"kubernetes.io/projected/17accd80-6cc2-4d3d-8e29-1db30b44852d-kube-api-access-2vzdb\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.384386 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.384362 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-home\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.384443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.384402 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17accd80-6cc2-4d3d-8e29-1db30b44852d-tls-certs\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.384554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.384534 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-dshm\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.396535 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.396514 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"openshift-service-ca.crt\"" Oct 31 03:32:07.446502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.446471 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"kube-root-ca.crt\"" Oct 31 03:32:07.446502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.446478 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"default-dockercfg-wznxn\"" Oct 31 03:32:07.446689 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.446500 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"simulator-w3-b98893-kserve-self-signed-certs\"" Oct 31 03:32:07.447929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.447908 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g"] Oct 31 03:32:07.485221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.485184 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-home\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.485432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.485281 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17accd80-6cc2-4d3d-8e29-1db30b44852d-tls-certs\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.485432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.485356 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-dshm\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.485432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.485405 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-model-cache\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.485615 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.485456 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-tmp-dir\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.485615 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.485480 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzdb\" (UniqueName: \"kubernetes.io/projected/17accd80-6cc2-4d3d-8e29-1db30b44852d-kube-api-access-2vzdb\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.485751 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.485725 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-home\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.486074 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.486051 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-tmp-dir\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.486313 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.486292 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-model-cache\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.494854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.494810 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-dshm\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.496718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.496694 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17accd80-6cc2-4d3d-8e29-1db30b44852d-tls-certs\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.550582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.550555 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzdb\" (UniqueName: \"kubernetes.io/projected/17accd80-6cc2-4d3d-8e29-1db30b44852d-kube-api-access-2vzdb\") pod \"simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:07.666626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:07.666541 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:08.061256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:08.061202 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g"] Oct 31 03:32:08.063636 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:32:08.063613 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17accd80_6cc2_4d3d_8e29_1db30b44852d.slice/crio-2b5c6966868f0ee6e8ce44a6db9dfa108500ad0cda9edc183506190fb8a55b05 WatchSource:0}: Error finding container 2b5c6966868f0ee6e8ce44a6db9dfa108500ad0cda9edc183506190fb8a55b05: Status 404 returned error can't find the container with id 2b5c6966868f0ee6e8ce44a6db9dfa108500ad0cda9edc183506190fb8a55b05 Oct 31 03:32:09.050972 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:09.050939 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" event={"ID":"17accd80-6cc2-4d3d-8e29-1db30b44852d","Type":"ContainerStarted","Data":"2b5c6966868f0ee6e8ce44a6db9dfa108500ad0cda9edc183506190fb8a55b05"} Oct 31 03:32:11.065053 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:11.065010 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" event={"ID":"17accd80-6cc2-4d3d-8e29-1db30b44852d","Type":"ContainerStarted","Data":"b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a"} Oct 31 03:32:11.182924 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:11.182876 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podStartSLOduration=1.997045915 podStartE2EDuration="4.182858768s" totalImagesPullingTime="2.185812853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:11.182823635 +0000 UTC m=+764.501566263" watchObservedRunningTime="2026-10-31 03:32:11.182858768 +0000 UTC m=+764.501601397" Oct 31 03:32:11.666827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:11.666787 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:17.667697 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:17.667665 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:17.680117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:17.680093 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:18.109405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:18.109373 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:32:19.161768 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:32:19.161733 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:32:21.209083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.209050 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5"] Oct 31 03:32:21.212729 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.212707 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.265432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.265409 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b63425-openshift-default-dockercfg-frs87\"" Oct 31 03:32:21.315107 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315076 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315135 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315274 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315314 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315368 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dbcd78-cf28-4850-935c-9ab453f1232f-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315395 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315575 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315420 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dvkw\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-kube-api-access-6dvkw\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315575 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315487 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.315575 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.315513 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.317362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.317340 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5"] Oct 31 03:32:21.416551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416525 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416577 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416601 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416621 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dbcd78-cf28-4850-935c-9ab453f1232f-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416638 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416652 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dvkw\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-kube-api-access-6dvkw\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416685 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416701 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.416993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.416753 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.417104 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.417016 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.417170 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.417093 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.417347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.417322 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.417452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.417399 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dbcd78-cf28-4850-935c-9ab453f1232f-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.417452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.417417 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.418907 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.418886 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.419133 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.419116 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.477788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.477735 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.477966 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.477951 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dvkw\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-kube-api-access-6dvkw\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.524927 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.524903 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:21.952424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.952402 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5"] Oct 31 03:32:21.954895 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:32:21.954866 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dbcd78_cf28_4850_935c_9ab453f1232f.slice/crio-1017f8e6d0cc30ab8538257ea7b35171e01301428e56cee4ee4f244a11b04b12 WatchSource:0}: Error finding container 1017f8e6d0cc30ab8538257ea7b35171e01301428e56cee4ee4f244a11b04b12: Status 404 returned error can't find the container with id 1017f8e6d0cc30ab8538257ea7b35171e01301428e56cee4ee4f244a11b04b12 Oct 31 03:32:21.957218 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.957189 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:32:21.957299 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.957283 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:32:21.957332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:21.957313 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:32:22.118488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:22.118437 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" event={"ID":"97dbcd78-cf28-4850-935c-9ab453f1232f","Type":"ContainerStarted","Data":"3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef"} Oct 31 03:32:22.118624 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:22.118521 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" event={"ID":"97dbcd78-cf28-4850-935c-9ab453f1232f","Type":"ContainerStarted","Data":"1017f8e6d0cc30ab8538257ea7b35171e01301428e56cee4ee4f244a11b04b12"} Oct 31 03:32:22.241435 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:22.241330 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" podStartSLOduration=1.241313466 podStartE2EDuration="1.241313466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:22.238652615 +0000 UTC m=+775.557395243" watchObservedRunningTime="2026-10-31 03:32:22.241313466 +0000 UTC m=+775.560056150" Oct 31 03:32:22.525904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:22.525823 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:23.526915 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:23.526880 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:32:23.527310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:23.526934 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:32:24.525673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:24.525639 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:32:24.525852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:24.525696 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:32:25.525444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:25.525412 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:32:25.525816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:25.525469 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:32:26.525518 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:26.525482 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:32:26.525984 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:26.525556 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:32:27.525961 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:27.525923 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:32:27.526395 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:27.526006 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:32:28.529058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:28.529022 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:28.529551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:28.529407 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:28.530451 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:28.530432 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:32:49.162117 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:32:49.162075 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:32:52.354349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.354319 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf"] Oct 31 03:32:52.360073 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.360057 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.423578 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.423542 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tmp-dir\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.423754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.423584 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-model-cache\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.423754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.423612 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlcz\" (UniqueName: \"kubernetes.io/projected/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-kube-api-access-zqlcz\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.423754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.423711 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-home\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.423754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.423741 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-dshm\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.423974 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.423923 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tls-certs\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.425753 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.425730 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"kube-root-ca.crt\"" Oct 31 03:32:52.425753 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.425732 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"openshift-service-ca.crt\"" Oct 31 03:32:52.432522 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.432504 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"default-dockercfg-f7wzm\"" Oct 31 03:32:52.465079 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.465055 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"simulator-w2-dec4a0-kserve-self-signed-certs\"" Oct 31 03:32:52.466691 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.466672 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf"] Oct 31 03:32:52.525075 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525040 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-model-cache\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525087 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlcz\" (UniqueName: \"kubernetes.io/projected/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-kube-api-access-zqlcz\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525141 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-home\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525176 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-dshm\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525409 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525269 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tls-certs\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525409 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525330 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tmp-dir\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525522 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-model-cache\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525606 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525533 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-home\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.525694 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.525676 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tmp-dir\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.527605 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.527580 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-dshm\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.527929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.527908 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tls-certs\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.587038 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.587004 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlcz\" (UniqueName: \"kubernetes.io/projected/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-kube-api-access-zqlcz\") pod \"simulator-w2-dec4a0-kserve-7c8885b858-db8bf\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:52.671525 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:52.671420 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:32:53.049359 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:53.049335 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf"] Oct 31 03:32:53.051967 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:32:53.051937 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d07ca2b_173c_4293_bc8f_2f8e5bf95e14.slice/crio-9ac8b830fcd6e97fe6192eeeab2f6820db1d2ebc881b8c2cbb5ea1fca9c7441d WatchSource:0}: Error finding container 9ac8b830fcd6e97fe6192eeeab2f6820db1d2ebc881b8c2cbb5ea1fca9c7441d: Status 404 returned error can't find the container with id 9ac8b830fcd6e97fe6192eeeab2f6820db1d2ebc881b8c2cbb5ea1fca9c7441d Oct 31 03:32:53.263260 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:53.263205 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" event={"ID":"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14","Type":"ContainerStarted","Data":"467e2c6bd7ea3fadc9b070f88b66ee11ca78a0422180b3da19a3cfea867af6d1"} Oct 31 03:32:53.263418 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:53.263272 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" event={"ID":"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14","Type":"ContainerStarted","Data":"9ac8b830fcd6e97fe6192eeeab2f6820db1d2ebc881b8c2cbb5ea1fca9c7441d"} Oct 31 03:32:53.382037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:53.381970 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podStartSLOduration=1.381951822 podStartE2EDuration="1.381951822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:53.380664445 +0000 UTC m=+806.699407075" watchObservedRunningTime="2026-10-31 03:32:53.381951822 +0000 UTC m=+806.700694450" Oct 31 03:32:53.672073 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:32:53.671996 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:33:02.671628 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:02.671591 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:33:02.683649 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:02.683622 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:33:03.325594 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:03.325567 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:33:19.164419 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:33:19.164384 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:33:21.969668 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:21.969628 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn"] Oct 31 03:33:21.973500 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:21.973477 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.026130 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.026107 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e00ad3-openshift-default-dockercfg-s9kbq\"" Oct 31 03:33:22.078389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.078357 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn"] Oct 31 03:33:22.112893 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.112857 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113039 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.112897 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113039 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.112965 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113039 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.113002 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gcf\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-kube-api-access-m9gcf\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.113089 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.113115 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68393835-77eb-4a74-99ca-4bded4593b56-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.113130 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113348 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.113222 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68393835-77eb-4a74-99ca-4bded4593b56-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.113348 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.113300 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.214938 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.214899 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215137 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.214953 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215137 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.214998 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215137 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215037 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gcf\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-kube-api-access-m9gcf\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215140 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215181 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68393835-77eb-4a74-99ca-4bded4593b56-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215206 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215268 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68393835-77eb-4a74-99ca-4bded4593b56-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215311 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215742 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215720 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.215932 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.215909 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.216136 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.216085 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68393835-77eb-4a74-99ca-4bded4593b56-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.216356 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.216338 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.216440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.216423 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.218150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.218122 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68393835-77eb-4a74-99ca-4bded4593b56-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.218539 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.218520 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.278735 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.278673 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.278960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.278936 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gcf\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-kube-api-access-m9gcf\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.286685 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.286661 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:22.735957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.735933 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn"] Oct 31 03:33:22.739058 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:33:22.739025 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68393835_77eb_4a74_99ca_4bded4593b56.slice/crio-6ef93d9e1b767cb3ebbc39580f9f58b8bc376fe2fdedbe3013f5da5c627c4f0f WatchSource:0}: Error finding container 6ef93d9e1b767cb3ebbc39580f9f58b8bc376fe2fdedbe3013f5da5c627c4f0f: Status 404 returned error can't find the container with id 6ef93d9e1b767cb3ebbc39580f9f58b8bc376fe2fdedbe3013f5da5c627c4f0f Oct 31 03:33:22.741222 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.741191 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:33:22.741329 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.741282 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:33:22.741389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:22.741327 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:33:23.417570 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:23.417536 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" event={"ID":"68393835-77eb-4a74-99ca-4bded4593b56","Type":"ContainerStarted","Data":"4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f"} Oct 31 03:33:23.417960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:23.417586 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" event={"ID":"68393835-77eb-4a74-99ca-4bded4593b56","Type":"ContainerStarted","Data":"6ef93d9e1b767cb3ebbc39580f9f58b8bc376fe2fdedbe3013f5da5c627c4f0f"} Oct 31 03:33:23.540705 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:23.540655 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" podStartSLOduration=2.540641155 podStartE2EDuration="2.540641155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:23.539509523 +0000 UTC m=+836.858252149" watchObservedRunningTime="2026-10-31 03:33:23.540641155 +0000 UTC m=+836.859383781" Oct 31 03:33:24.287591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:24.287554 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:24.289166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:24.289135 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:33:24.289381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:24.289178 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:33:25.287255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:25.287199 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:33:25.287715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:25.287690 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:33:26.287716 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:26.287685 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:33:26.288090 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:26.287736 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:33:27.287874 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:27.287841 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:33:27.288308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:27.287894 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:33:28.287827 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:28.287786 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:33:28.287981 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:28.287856 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:33:29.291665 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:29.291633 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:29.292031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:29.291882 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:29.292606 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:29.292592 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:33:49.166751 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:33:49.166658 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:33:51.031737 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.031702 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h"] Oct 31 03:33:51.035192 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.035171 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.136237 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.136208 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-tmp-dir\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.136403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.136249 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhff\" (UniqueName: \"kubernetes.io/projected/7161bd91-908a-4cee-8195-7fa4ca72d836-kube-api-access-lqhff\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.136403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.136271 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7161bd91-908a-4cee-8195-7fa4ca72d836-tls-certs\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.136403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.136348 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-model-cache\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.136403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.136375 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-home\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.136546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.136413 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-dshm\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.137737 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.137712 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"default-dockercfg-jm4bw\"" Oct 31 03:33:51.137876 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.137819 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"premium-w4-096812-kserve-self-signed-certs\"" Oct 31 03:33:51.137876 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.137820 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"openshift-service-ca.crt\"" Oct 31 03:33:51.137973 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.137817 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"kube-root-ca.crt\"" Oct 31 03:33:51.139479 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.139460 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h"] Oct 31 03:33:51.236997 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.236966 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-tmp-dir\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237004 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhff\" (UniqueName: \"kubernetes.io/projected/7161bd91-908a-4cee-8195-7fa4ca72d836-kube-api-access-lqhff\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237034 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7161bd91-908a-4cee-8195-7fa4ca72d836-tls-certs\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237070 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-model-cache\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237110 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-home\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237145 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-dshm\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237504 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237454 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-tmp-dir\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237504 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237488 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-model-cache\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.237627 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.237609 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-home\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.239469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.239444 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-dshm\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.240104 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.240089 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7161bd91-908a-4cee-8195-7fa4ca72d836-tls-certs\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.298528 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.298451 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhff\" (UniqueName: \"kubernetes.io/projected/7161bd91-908a-4cee-8195-7fa4ca72d836-kube-api-access-lqhff\") pod \"premium-w4-096812-kserve-7c54dd64c6-xm75h\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.346936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.346894 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:33:51.727887 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:51.727861 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h"] Oct 31 03:33:51.729724 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:33:51.729696 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7161bd91_908a_4cee_8195_7fa4ca72d836.slice/crio-e115dfee0635f864c450322405c73cc4ea46258d7e47f66a90d94256039c3b42 WatchSource:0}: Error finding container e115dfee0635f864c450322405c73cc4ea46258d7e47f66a90d94256039c3b42: Status 404 returned error can't find the container with id e115dfee0635f864c450322405c73cc4ea46258d7e47f66a90d94256039c3b42 Oct 31 03:33:52.577446 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:52.577408 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" event={"ID":"7161bd91-908a-4cee-8195-7fa4ca72d836","Type":"ContainerStarted","Data":"750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8"} Oct 31 03:33:52.577821 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:52.577464 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" event={"ID":"7161bd91-908a-4cee-8195-7fa4ca72d836","Type":"ContainerStarted","Data":"e115dfee0635f864c450322405c73cc4ea46258d7e47f66a90d94256039c3b42"} Oct 31 03:33:52.695672 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:52.695621 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podStartSLOduration=2.695607835 podStartE2EDuration="2.695607835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:52.69389535 +0000 UTC m=+866.012637979" watchObservedRunningTime="2026-10-31 03:33:52.695607835 +0000 UTC m=+866.014350461" Oct 31 03:33:53.347645 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:33:53.347617 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:34:01.347353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:01.347318 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:34:01.359816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:01.359782 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:34:01.646908 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:01.646831 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:34:10.350286 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.350252 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6"] Oct 31 03:34:10.353661 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.353645 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.414175 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.414143 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-model-cache\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.414331 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.414197 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37a6959-4519-4d12-aa29-86651e36705b-tls-certs\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.414331 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.414282 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmmq\" (UniqueName: \"kubernetes.io/projected/b37a6959-4519-4d12-aa29-86651e36705b-kube-api-access-pgmmq\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.414331 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.414319 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-dshm\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.414455 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.414435 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-tmp-dir\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.414489 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.414466 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-home\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.458291 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.458265 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"premium-w2-dec4a0-kserve-self-signed-certs\"" Oct 31 03:34:10.460036 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.460012 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6"] Oct 31 03:34:10.515025 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.514990 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-tmp-dir\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515038 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-home\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515091 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-model-cache\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515397 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515271 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37a6959-4519-4d12-aa29-86651e36705b-tls-certs\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515397 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515333 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmmq\" (UniqueName: \"kubernetes.io/projected/b37a6959-4519-4d12-aa29-86651e36705b-kube-api-access-pgmmq\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515397 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515365 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-dshm\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515504 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-tmp-dir\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515552 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-home\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.515642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.515619 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-model-cache\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.517619 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.517596 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-dshm\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.517984 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.517966 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37a6959-4519-4d12-aa29-86651e36705b-tls-certs\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.576923 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.576888 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmmq\" (UniqueName: \"kubernetes.io/projected/b37a6959-4519-4d12-aa29-86651e36705b-kube-api-access-pgmmq\") pod \"premium-w2-dec4a0-kserve-7d884d7b5f-58ms6\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:10.665967 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:10.665878 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:11.108877 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:11.108844 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6"] Oct 31 03:34:11.110617 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:34:11.110590 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37a6959_4519_4d12_aa29_86651e36705b.slice/crio-630329d940b62a16261f74ae8b373ceefe635ab9cc46aae7cc198e81c0506404 WatchSource:0}: Error finding container 630329d940b62a16261f74ae8b373ceefe635ab9cc46aae7cc198e81c0506404: Status 404 returned error can't find the container with id 630329d940b62a16261f74ae8b373ceefe635ab9cc46aae7cc198e81c0506404 Oct 31 03:34:11.695993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:11.695954 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" event={"ID":"b37a6959-4519-4d12-aa29-86651e36705b","Type":"ContainerStarted","Data":"a92f1ef95f3c0243d85fa8c9612a111d07b96b2fdfebb43802ed8fbeebaaba18"} Oct 31 03:34:11.696456 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:11.696005 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" event={"ID":"b37a6959-4519-4d12-aa29-86651e36705b","Type":"ContainerStarted","Data":"630329d940b62a16261f74ae8b373ceefe635ab9cc46aae7cc198e81c0506404"} Oct 31 03:34:12.665983 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:12.665951 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:19.163168 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:34:19.163134 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:34:20.666722 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:20.666685 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:20.679150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:20.679122 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:20.750596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:20.750573 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:34:20.798107 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:20.798059 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podStartSLOduration=10.798045914 podStartE2EDuration="10.798045914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:11.820606566 +0000 UTC m=+885.139349195" watchObservedRunningTime="2026-10-31 03:34:20.798045914 +0000 UTC m=+894.116788545" Oct 31 03:34:22.870163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.870126 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg"] Oct 31 03:34:22.873897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.873881 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.928852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.928825 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-096812-openshift-default-dockercfg-7vgck\"" Oct 31 03:34:22.941298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941258 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941305 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941342 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941369 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941439 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941469 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941731 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941486 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941731 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941512 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l697w\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-kube-api-access-l697w\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.941731 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.941533 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:22.980265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:22.980221 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg"] Oct 31 03:34:23.042665 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042639 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.042814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042674 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.042814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042701 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.042814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042719 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.042814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042754 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.042814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042774 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.042814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042788 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.043056 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042814 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l697w\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-kube-api-access-l697w\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.043056 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.042844 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.043163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.043098 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.043362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.043335 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.043459 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.043374 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.043508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.043449 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.043731 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.043713 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.045170 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.045145 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.045422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.045403 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.125795 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.125732 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.146795 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.146768 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l697w\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-kube-api-access-l697w\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.181462 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.181435 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq"] Oct 31 03:34:23.187012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.186990 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.188790 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.188774 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:23.241658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.241632 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b98893-openshift-default-dockercfg-f7nhz\"" Oct 31 03:34:23.244818 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.244786 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.244950 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.244824 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqw2c\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-kube-api-access-hqw2c\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.244950 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.244854 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.244950 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.244896 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.244950 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.244934 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b60600-b26d-4594-82aa-0f45f55b8907-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.245190 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.244958 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.245190 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.244991 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.245190 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.245124 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.245352 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.245212 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.249868 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.249796 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq"] Oct 31 03:34:23.347963 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.347929 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.347980 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348002 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b60600-b26d-4594-82aa-0f45f55b8907-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348027 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348054 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348083 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348111 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348238 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348265 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqw2c\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-kube-api-access-hqw2c\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348574 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348546 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348597 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348655 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.348833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.348784 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.349058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.349033 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b60600-b26d-4594-82aa-0f45f55b8907-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.350538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.350522 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.350686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.350669 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.409071 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.409013 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.409284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.409266 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqw2c\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-kube-api-access-hqw2c\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.502092 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.502063 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:23.613033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.613007 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg"] Oct 31 03:34:23.614763 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:34:23.614742 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f391f7_fcac_4181_aeb6_11a9b5e27eb0.slice/crio-4745f068045599001d27fa5b1d45fa0a4ba06f300ced55f0e4d29358614cb3fd WatchSource:0}: Error finding container 4745f068045599001d27fa5b1d45fa0a4ba06f300ced55f0e4d29358614cb3fd: Status 404 returned error can't find the container with id 4745f068045599001d27fa5b1d45fa0a4ba06f300ced55f0e4d29358614cb3fd Oct 31 03:34:23.616983 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.616952 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:34:23.617058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.617016 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:34:23.617058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.617043 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:34:23.758958 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.758924 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" event={"ID":"50f391f7-fcac-4181-aeb6-11a9b5e27eb0","Type":"ContainerStarted","Data":"2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780"} Oct 31 03:34:23.759110 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.758986 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" event={"ID":"50f391f7-fcac-4181-aeb6-11a9b5e27eb0","Type":"ContainerStarted","Data":"4745f068045599001d27fa5b1d45fa0a4ba06f300ced55f0e4d29358614cb3fd"} Oct 31 03:34:23.877846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.877738 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq"] Oct 31 03:34:23.880693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.880636 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podStartSLOduration=1.880618908 podStartE2EDuration="1.880618908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:23.878931266 +0000 UTC m=+897.197673894" watchObservedRunningTime="2026-10-31 03:34:23.880618908 +0000 UTC m=+897.199361537" Oct 31 03:34:23.883286 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.883194 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:34:23.883392 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.883293 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:34:23.883392 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:23.883325 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:34:24.190683 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:24.190596 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:24.192351 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:24.192329 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:34:24.192414 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:24.192382 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:34:24.805251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:24.797365 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" event={"ID":"f9b60600-b26d-4594-82aa-0f45f55b8907","Type":"ContainerStarted","Data":"8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe"} Oct 31 03:34:24.805251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:24.797424 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" event={"ID":"f9b60600-b26d-4594-82aa-0f45f55b8907","Type":"ContainerStarted","Data":"a28c2d51aba99a99f28b9751f7df8359a7dd8b1a5336560a8871addf021fb1f0"} Oct 31 03:34:24.918937 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:24.918876 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" podStartSLOduration=1.918856981 podStartE2EDuration="1.918856981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:24.918132579 +0000 UTC m=+898.236875211" watchObservedRunningTime="2026-10-31 03:34:24.918856981 +0000 UTC m=+898.237599624" Oct 31 03:34:25.197685 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:25.197649 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:34:25.197983 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:25.197944 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:34:25.503219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:25.503127 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:25.503819 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:25.503798 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 03:34:25.503881 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:25.503841 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 03:34:26.192253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:26.191643 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:34:26.192253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:26.191705 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:34:26.502673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:26.502585 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 03:34:26.502907 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:26.502873 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 03:34:26.948421 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:26.948395 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:34:26.949411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:26.949393 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:34:27.197252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:27.194632 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:34:27.197252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:27.194706 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:34:27.504248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:27.502566 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 03:34:27.504248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:27.502625 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 03:34:28.192280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:28.191588 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:34:28.192280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:28.191643 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:34:28.509307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:28.504572 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 03:34:28.509307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:28.504631 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 03:34:29.191608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:29.191573 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:34:29.191916 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:29.191887 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:34:29.502897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:29.502814 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 03:34:29.502897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:29.502878 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 03:34:30.197812 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:30.197773 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:30.198520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:30.198495 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:30.199574 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:30.199555 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:34:30.507430 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:30.507359 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:30.507661 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:30.507645 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:30.508305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:34:30.508290 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 03:34:49.163441 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:34:49.163404 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:35:19.162834 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:35:19.162799 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:35:24.238024 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.237992 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z"] Oct 31 03:35:24.241811 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.241794 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.294985 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.294948 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-14cb5a-openshift-default-dockercfg-hg7j6\"" Oct 31 03:35:24.336720 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.336682 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.336910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.336770 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.336910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.336856 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.336910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.336886 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.337078 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.337005 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.337078 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.337030 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fz2w\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-kube-api-access-4fz2w\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.337078 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.337049 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.337269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.337136 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45a1814-9960-4ad4-9f1f-4991f4558d02-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.337269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.337163 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.346813 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.346786 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z"] Oct 31 03:35:24.438001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.437966 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438188 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438023 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438188 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438055 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438206 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438266 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fz2w\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-kube-api-access-4fz2w\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438287 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438328 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45a1814-9960-4ad4-9f1f-4991f4558d02-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438357 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438433 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438448 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438468 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438549 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438702 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.438965 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.438944 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45a1814-9960-4ad4-9f1f-4991f4558d02-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.440713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.440689 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.441813 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.441791 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.500445 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.500382 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.500577 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.500556 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fz2w\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-kube-api-access-4fz2w\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.554389 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.554360 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:24.988518 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.988493 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z"] Oct 31 03:35:24.989838 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:35:24.989810 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45a1814_9960_4ad4_9f1f_4991f4558d02.slice/crio-91d2940f92f70506ed9742c04e1cf941edafe2b750643eed80af76c1b621d494 WatchSource:0}: Error finding container 91d2940f92f70506ed9742c04e1cf941edafe2b750643eed80af76c1b621d494: Status 404 returned error can't find the container with id 91d2940f92f70506ed9742c04e1cf941edafe2b750643eed80af76c1b621d494 Oct 31 03:35:24.992371 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.992335 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:35:24.992443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.992420 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:35:24.992484 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:24.992469 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:35:25.119833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:25.119794 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" event={"ID":"a45a1814-9960-4ad4-9f1f-4991f4558d02","Type":"ContainerStarted","Data":"a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8"} Oct 31 03:35:25.119973 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:25.119857 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" event={"ID":"a45a1814-9960-4ad4-9f1f-4991f4558d02","Type":"ContainerStarted","Data":"91d2940f92f70506ed9742c04e1cf941edafe2b750643eed80af76c1b621d494"} Oct 31 03:35:25.242573 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:25.242481 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podStartSLOduration=1.242462408 podStartE2EDuration="1.242462408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:25.240517074 +0000 UTC m=+958.559259701" watchObservedRunningTime="2026-10-31 03:35:25.242462408 +0000 UTC m=+958.561205036" Oct 31 03:35:25.554507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:25.554419 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:26.577919 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:26.577881 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:35:26.578544 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:26.578512 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 03:35:27.561254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:27.560624 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:35:27.561254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:27.560688 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 03:35:28.557266 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:28.555500 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:35:28.557266 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:28.555556 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 03:35:29.574251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:29.565560 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:35:29.574251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:29.565618 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 03:35:30.556639 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:30.556607 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:35:30.556907 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:30.556879 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 03:35:31.702320 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:31.702286 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:31.702790 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:31.702772 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:31.703581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:35:31.703559 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:35:49.175983 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:35:49.175915 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:36:08.334721 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.334683 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6"] Oct 31 03:36:08.338812 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.338789 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393059 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393031 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393059 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393063 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393243 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393081 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pmt\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-kube-api-access-79pmt\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393243 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393153 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393243 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393215 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393281 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393310 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393346 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.393503 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.393379 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.442084 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.442054 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6"] Oct 31 03:36:08.494271 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494242 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494289 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494322 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494347 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494389 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494483 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494509 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494539 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pmt\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-kube-api-access-79pmt\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494579 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494729 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494778 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.494998 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.494970 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.495429 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.495344 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.495429 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.495346 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.497021 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.496999 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.497132 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.497060 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.555719 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.555695 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.555836 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.555786 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pmt\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-kube-api-access-79pmt\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:08.651257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:08.651179 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:09.079375 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.079344 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6"] Oct 31 03:36:09.083379 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:36:09.083340 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b69ec0a_4977_4db7_bab5_bc5ca4a808d4.slice/crio-544aed8cf112f0e404a89f6fdb6d49370ddd2d219a96950c68c8692ed7e8e279 WatchSource:0}: Error finding container 544aed8cf112f0e404a89f6fdb6d49370ddd2d219a96950c68c8692ed7e8e279: Status 404 returned error can't find the container with id 544aed8cf112f0e404a89f6fdb6d49370ddd2d219a96950c68c8692ed7e8e279 Oct 31 03:36:09.085909 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.085876 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:36:09.085995 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.085944 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:36:09.085995 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.085975 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:36:09.290248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.290029 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7"] Oct 31 03:36:09.295323 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.295292 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.347900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.347837 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-dec4a0-openshift-default-dockercfg-5rrgk\"" Oct 31 03:36:09.359309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.359282 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7"] Oct 31 03:36:09.403060 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403023 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403165 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403202 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916dada2-c7fc-4667-8682-fabab974e046-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403248 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403286 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403307 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403331 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6dzq\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-kube-api-access-z6dzq\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403355 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916dada2-c7fc-4667-8682-fabab974e046-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.403556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.403413 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.404473 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.404448 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" event={"ID":"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4","Type":"ContainerStarted","Data":"f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a"} Oct 31 03:36:09.404606 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.404496 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" event={"ID":"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4","Type":"ContainerStarted","Data":"544aed8cf112f0e404a89f6fdb6d49370ddd2d219a96950c68c8692ed7e8e279"} Oct 31 03:36:09.504356 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504322 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504363 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504389 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6dzq\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-kube-api-access-z6dzq\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504413 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916dada2-c7fc-4667-8682-fabab974e046-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504473 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504564 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504696 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504732 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916dada2-c7fc-4667-8682-fabab974e046-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.504805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.504769 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.505946 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.505191 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.505946 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.505258 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.508296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.507300 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916dada2-c7fc-4667-8682-fabab974e046-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.508743 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.508717 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.508942 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.508925 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.509065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.508925 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916dada2-c7fc-4667-8682-fabab974e046-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.512004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.511981 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.540189 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.539993 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podStartSLOduration=1.5399737500000001 podStartE2EDuration="1.53997375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:09.537022866 +0000 UTC m=+1002.855765493" watchObservedRunningTime="2026-10-31 03:36:09.53997375 +0000 UTC m=+1002.858716379" Oct 31 03:36:09.569266 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.566274 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6dzq\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-kube-api-access-z6dzq\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.569266 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.568879 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.612343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.612215 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:09.688314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:09.673439 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:10.277342 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.277243 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7"] Oct 31 03:36:10.286269 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:36:10.286026 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916dada2_c7fc_4667_8682_fabab974e046.slice/crio-b4921d6a67812ca78d18ffd5b7d2cb8164f77d8333e4bd356cbba812346bd92f WatchSource:0}: Error finding container b4921d6a67812ca78d18ffd5b7d2cb8164f77d8333e4bd356cbba812346bd92f: Status 404 returned error can't find the container with id b4921d6a67812ca78d18ffd5b7d2cb8164f77d8333e4bd356cbba812346bd92f Oct 31 03:36:10.289104 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.289064 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:36:10.289279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.289167 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:36:10.289279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.289249 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:36:10.419016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.418975 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" event={"ID":"916dada2-c7fc-4667-8682-fabab974e046","Type":"ContainerStarted","Data":"21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290"} Oct 31 03:36:10.419495 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.419078 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" event={"ID":"916dada2-c7fc-4667-8682-fabab974e046","Type":"ContainerStarted","Data":"b4921d6a67812ca78d18ffd5b7d2cb8164f77d8333e4bd356cbba812346bd92f"} Oct 31 03:36:10.560419 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.553955 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podStartSLOduration=1.553935646 podStartE2EDuration="1.553935646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:10.550592524 +0000 UTC m=+1003.869335151" watchObservedRunningTime="2026-10-31 03:36:10.553935646 +0000 UTC m=+1003.872678274" Oct 31 03:36:10.613033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.612994 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:10.662252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.657317 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:36:10.662252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:10.657371 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 03:36:11.614508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:11.614464 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:36:11.614915 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:11.614515 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 03:36:11.651893 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:11.651860 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:36:11.652046 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:11.651909 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 03:36:12.615668 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:12.615622 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:36:12.616196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:12.615683 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 03:36:12.652453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:12.652423 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:36:12.652587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:12.652478 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 03:36:13.612662 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:13.612628 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:36:13.620506 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:13.612850 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 03:36:13.651581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:13.651552 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:36:13.651715 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:13.651604 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 03:36:14.618664 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:14.618629 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:36:14.618994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:14.618967 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 03:36:14.725712 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:14.725677 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:36:14.726331 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:14.726289 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 03:36:15.125342 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:15.125312 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp"] Oct 31 03:36:15.125974 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:15.125818 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" containerID="cri-o://925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311" gracePeriod=30 Oct 31 03:36:15.612981 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:15.612950 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:36:15.613167 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:15.613010 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 03:36:15.651728 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:15.651691 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:36:15.651863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:15.651765 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 03:36:16.612686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:16.612636 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:36:16.613191 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:16.612714 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 03:36:16.925140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:16.925046 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:16.925505 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:16.925478 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:16.926527 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:16.926506 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:36:17.617360 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:17.617330 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:17.617735 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:17.617597 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:17.618200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:17.618184 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:36:19.163437 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:36:19.163369 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:36:20.385545 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.385519 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:36:20.439654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439594 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-envoy\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.439786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439697 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-kube-api-access-dr7q7\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-kube-api-access-dr7q7\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.439786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439729 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-podinfo\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.439864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439808 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-certs\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.439864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439849 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-credential-socket\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.439934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439885 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-data\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.439934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439923 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-socket\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.440029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.439974 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istiod-ca-cert\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.440082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440031 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-token\") pod \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\" (UID: \"9b9fe54c-889b-474c-94a1-20736e7dbe6d\") " Oct 31 03:36:20.440178 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440105 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-certs" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:20.440299 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440218 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-data" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:20.440377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440355 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-socket" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:20.440452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440324 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-credential-socket" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:20.440600 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440569 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istiod-ca-cert" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:20.440703 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440685 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.440765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440711 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.440765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440725 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.440765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440739 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.440765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.440752 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.441969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.441942 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-envoy" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:20.442094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.441994 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-podinfo" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:36:20.442094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.442082 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-kube-api-access-dr7q7" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "kube-api-access-dr7q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:20.442556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.442535 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-token" pod "9b9fe54c-889b-474c-94a1-20736e7dbe6d" (UID: "9b9fe54c-889b-474c-94a1-20736e7dbe6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:20.484283 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.484253 4934 generic.go:356] "Generic (PLEG): container finished" podID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerID="925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311" exitCode=0 Oct 31 03:36:20.484377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.484302 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" Oct 31 03:36:20.484377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.484337 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" event={"ID":"9b9fe54c-889b-474c-94a1-20736e7dbe6d","Type":"ContainerDied","Data":"925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311"} Oct 31 03:36:20.484444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.484393 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp" event={"ID":"9b9fe54c-889b-474c-94a1-20736e7dbe6d","Type":"ContainerDied","Data":"23e556c78238b866430696705ef800e06cc9572f04c376da73ebbc52ff250350"} Oct 31 03:36:20.484444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.484420 4934 scope.go:122] "RemoveContainer" containerID="925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311" Oct 31 03:36:20.493271 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.493252 4934 scope.go:122] "RemoveContainer" containerID="925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311" Oct 31 03:36:20.493530 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:36:20.493513 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311\": container with ID starting with 925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311 not found: ID does not exist" containerID="925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311" Oct 31 03:36:20.493573 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.493539 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311"} err="failed to get container status \"925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311\": rpc error: code = NotFound desc = could not find container \"925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311\": container with ID starting with 925df9b065dece8dcab83f7192c4559af188b29061423486600384da084dc311 not found: ID does not exist" Oct 31 03:36:20.541250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.541205 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr7q7\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-kube-api-access-dr7q7\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.541250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.541238 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.541250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.541249 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.541250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.541258 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fe54c-889b-474c-94a1-20736e7dbe6d-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:20.670282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.670255 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp"] Oct 31 03:36:20.674758 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.674735 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-zphzp"] Oct 31 03:36:20.821048 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:20.821013 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" path="/var/lib/kubelet/pods/9b9fe54c-889b-474c-94a1-20736e7dbe6d/volumes" Oct 31 03:36:49.162497 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:36:49.162418 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:36:59.708854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.708823 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s"] Oct 31 03:36:59.709304 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.709289 4934 cpu_manager.go:414] "containerMap: removing container" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" Oct 31 03:36:59.709365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.709307 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" Oct 31 03:36:59.709417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.709406 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9fe54c-889b-474c-94a1-20736e7dbe6d" containerName="istio-proxy" Oct 31 03:36:59.712565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.712550 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.769268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.768714 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"openshift-service-ca.crt\"" Oct 31 03:36:59.815102 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.815065 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"default-dockercfg-5rnfw\"" Oct 31 03:36:59.815102 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.815089 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"kube-root-ca.crt\"" Oct 31 03:36:59.815314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.815145 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"distinct-w6-b63425-kserve-self-signed-certs\"" Oct 31 03:36:59.817216 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.817198 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s"] Oct 31 03:36:59.856608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.856072 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tls-certs\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.856608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.856137 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-home\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.856608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.856213 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tmp-dir\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.856608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.856286 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4rj\" (UniqueName: \"kubernetes.io/projected/d3a4cda6-f9c8-441c-ad18-c335c59fb134-kube-api-access-fh4rj\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.856608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.856340 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-model-cache\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.856608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.856455 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-dshm\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.957710 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.957664 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-model-cache\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.957900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.957821 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-dshm\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.957961 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.957911 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tls-certs\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.957961 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.957947 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-home\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.958072 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.958023 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tmp-dir\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.958140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.958103 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-model-cache\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.958192 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.958152 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4rj\" (UniqueName: \"kubernetes.io/projected/d3a4cda6-f9c8-441c-ad18-c335c59fb134-kube-api-access-fh4rj\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.958819 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.958794 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-home\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.959593 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.959526 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tmp-dir\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.961089 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.961060 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-dshm\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:36:59.961330 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:36:59.961312 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tls-certs\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:37:00.019398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:00.019368 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4rj\" (UniqueName: \"kubernetes.io/projected/d3a4cda6-f9c8-441c-ad18-c335c59fb134-kube-api-access-fh4rj\") pod \"distinct-w6-b63425-kserve-56c46df849-vtl8s\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:37:00.024115 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:00.024090 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:37:00.449222 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:00.449201 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s"] Oct 31 03:37:00.451202 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:37:00.451174 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3a4cda6_f9c8_441c_ad18_c335c59fb134.slice/crio-c1d2d01a6962bf0cb0d400f524014269b0dab3a065e38edbc59305ec1b5bd82c WatchSource:0}: Error finding container c1d2d01a6962bf0cb0d400f524014269b0dab3a065e38edbc59305ec1b5bd82c: Status 404 returned error can't find the container with id c1d2d01a6962bf0cb0d400f524014269b0dab3a065e38edbc59305ec1b5bd82c Oct 31 03:37:00.737411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:00.737333 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" event={"ID":"d3a4cda6-f9c8-441c-ad18-c335c59fb134","Type":"ContainerStarted","Data":"0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb"} Oct 31 03:37:00.737411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:00.737371 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" event={"ID":"d3a4cda6-f9c8-441c-ad18-c335c59fb134","Type":"ContainerStarted","Data":"c1d2d01a6962bf0cb0d400f524014269b0dab3a065e38edbc59305ec1b5bd82c"} Oct 31 03:37:00.854905 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:00.854852 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podStartSLOduration=1.8548345689999999 podStartE2EDuration="1.854834569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:00.854259811 +0000 UTC m=+1054.173002440" watchObservedRunningTime="2026-10-31 03:37:00.854834569 +0000 UTC m=+1054.173577197" Oct 31 03:37:01.024357 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:01.024268 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:37:03.080947 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.080912 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc"] Oct 31 03:37:03.084700 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.084684 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.230009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.229977 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"distinct-w3-b98893-kserve-self-signed-certs\"" Oct 31 03:37:03.230322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.230304 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc"] Oct 31 03:37:03.273247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.268527 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-tmp-dir\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.273247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.268694 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-home\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.273247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.268755 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4a9ca-e8db-4587-9a1a-08471c508625-tls-certs\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.273247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.268856 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-model-cache\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.273247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.271400 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl59d\" (UniqueName: \"kubernetes.io/projected/03f4a9ca-e8db-4587-9a1a-08471c508625-kube-api-access-cl59d\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.273247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.271547 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-dshm\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.373090 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.373006 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-model-cache\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.373376 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.373353 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl59d\" (UniqueName: \"kubernetes.io/projected/03f4a9ca-e8db-4587-9a1a-08471c508625-kube-api-access-cl59d\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.373572 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.373556 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-dshm\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.373717 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.373701 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-tmp-dir\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.373848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.373835 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-home\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.373959 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.373946 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4a9ca-e8db-4587-9a1a-08471c508625-tls-certs\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.374126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.374003 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-tmp-dir\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.374265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.373570 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-model-cache\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.374707 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.374684 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-home\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.377166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.377121 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4a9ca-e8db-4587-9a1a-08471c508625-tls-certs\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.377864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.377834 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-dshm\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.457957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.457891 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl59d\" (UniqueName: \"kubernetes.io/projected/03f4a9ca-e8db-4587-9a1a-08471c508625-kube-api-access-cl59d\") pod \"distinct-w3-b98893-kserve-75fc645c4c-r4htc\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:03.702300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:03.701435 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:04.264116 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:04.264083 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc"] Oct 31 03:37:04.273705 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:37:04.273675 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03f4a9ca_e8db_4587_9a1a_08471c508625.slice/crio-28d6ed7827a1a58cce11e4c9680b065bc92d7a4b7c5f4e7f8111390a2e3403e8 WatchSource:0}: Error finding container 28d6ed7827a1a58cce11e4c9680b065bc92d7a4b7c5f4e7f8111390a2e3403e8: Status 404 returned error can't find the container with id 28d6ed7827a1a58cce11e4c9680b065bc92d7a4b7c5f4e7f8111390a2e3403e8 Oct 31 03:37:04.791524 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:04.791495 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" event={"ID":"03f4a9ca-e8db-4587-9a1a-08471c508625","Type":"ContainerStarted","Data":"5d6eaf2bd878d9f5364b1c74815a5bb13d0f617965afefc2455fe7ab4d7d3064"} Oct 31 03:37:04.791698 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:04.791531 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" event={"ID":"03f4a9ca-e8db-4587-9a1a-08471c508625","Type":"ContainerStarted","Data":"28d6ed7827a1a58cce11e4c9680b065bc92d7a4b7c5f4e7f8111390a2e3403e8"} Oct 31 03:37:04.909477 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:04.909431 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podStartSLOduration=1.9094170529999999 podStartE2EDuration="1.909417053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:04.908067281 +0000 UTC m=+1058.226809903" watchObservedRunningTime="2026-10-31 03:37:04.909417053 +0000 UTC m=+1058.228159678" Oct 31 03:37:05.702506 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:05.702481 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:10.025303 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:10.025269 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:37:10.039521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:10.039494 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:37:10.833493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:10.833468 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:37:13.702871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:13.702835 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:13.715150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:13.715123 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:13.849125 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:13.849104 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:37:19.162347 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:37:19.162313 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:37:23.942693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:23.942662 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz"] Oct 31 03:37:23.946374 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:23.946356 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098124 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098340 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098171 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098340 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098300 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098382 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098524 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098661 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098614 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098662 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45eb878-d4e7-4685-866c-3c4027aa4130-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098872 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098840 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.098948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.098907 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5c2v\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-kube-api-access-l5c2v\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.188493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.188466 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz"] Oct 31 03:37:24.200034 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.199973 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200034 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200007 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200034 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200026 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45eb878-d4e7-4685-866c-3c4027aa4130-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200048 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200099 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5c2v\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-kube-api-access-l5c2v\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200153 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200179 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200243 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200552 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200318 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200552 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200418 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200656 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200612 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200656 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200629 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.200861 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.200843 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.201180 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.201160 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45eb878-d4e7-4685-866c-3c4027aa4130-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.203138 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.203117 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.203254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.203116 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.261158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.261134 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.261311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.261294 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5c2v\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-kube-api-access-l5c2v\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.559416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.559391 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:24.990124 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.990100 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz"] Oct 31 03:37:24.992064 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:37:24.992038 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45eb878_d4e7_4685_866c_3c4027aa4130.slice/crio-1f9941cee97a7839cfd91f1dd64bf24776fd4d8602d3042577e980489baf0d94 WatchSource:0}: Error finding container 1f9941cee97a7839cfd91f1dd64bf24776fd4d8602d3042577e980489baf0d94: Status 404 returned error can't find the container with id 1f9941cee97a7839cfd91f1dd64bf24776fd4d8602d3042577e980489baf0d94 Oct 31 03:37:24.994212 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.994185 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:37:24.994292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.994263 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:37:24.994330 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:24.994292 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:37:25.173480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.173450 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc"] Oct 31 03:37:25.177841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.177813 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.250329 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.247498 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc"] Oct 31 03:37:25.313571 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313539 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313579 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313676 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e14a0a-db97-451a-b942-171d5d643415-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313822 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313742 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313822 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313780 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrb7f\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-kube-api-access-jrb7f\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313822 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313801 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313918 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313897 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e14a0a-db97-451a-b942-171d5d643415-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313961 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313947 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.313990 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.313967 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415019 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.414981 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e14a0a-db97-451a-b942-171d5d643415-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415073 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415102 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415184 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415205 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415271 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e14a0a-db97-451a-b942-171d5d643415-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415315 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415350 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrb7f\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-kube-api-access-jrb7f\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415378 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415653 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415557 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415714 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415682 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415741 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.415860 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.415838 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e14a0a-db97-451a-b942-171d5d643415-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.416150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.416072 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.417823 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.417800 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.417942 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.417879 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e14a0a-db97-451a-b942-171d5d643415-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.486252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.484028 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.489864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.489833 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrb7f\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-kube-api-access-jrb7f\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.491713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.491691 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:25.898658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.898623 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" event={"ID":"f45eb878-d4e7-4685-866c-3c4027aa4130","Type":"ContainerStarted","Data":"b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067"} Oct 31 03:37:25.898833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.898684 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" event={"ID":"f45eb878-d4e7-4685-866c-3c4027aa4130","Type":"ContainerStarted","Data":"1f9941cee97a7839cfd91f1dd64bf24776fd4d8602d3042577e980489baf0d94"} Oct 31 03:37:25.928498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.928478 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc"] Oct 31 03:37:25.930726 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:37:25.930706 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e14a0a_db97_451a_b942_171d5d643415.slice/crio-a8a52d7ca19ed067a89b08355fec62b4d0aa5ee149af0e38dcd6f203fbbb41ee WatchSource:0}: Error finding container a8a52d7ca19ed067a89b08355fec62b4d0aa5ee149af0e38dcd6f203fbbb41ee: Status 404 returned error can't find the container with id a8a52d7ca19ed067a89b08355fec62b4d0aa5ee149af0e38dcd6f203fbbb41ee Oct 31 03:37:25.932880 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.932850 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:37:25.932977 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.932916 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:37:25.933036 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:25.932979 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:37:26.021824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:26.021720 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podStartSLOduration=3.021702044 podStartE2EDuration="3.021702044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:26.019189694 +0000 UTC m=+1079.337932322" watchObservedRunningTime="2026-10-31 03:37:26.021702044 +0000 UTC m=+1079.340444670" Oct 31 03:37:26.571253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:26.564134 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:26.571253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:26.564282 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:37:26.571253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:26.564324 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 03:37:26.917316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:26.916955 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" event={"ID":"a2e14a0a-db97-451a-b942-171d5d643415","Type":"ContainerStarted","Data":"9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e"} Oct 31 03:37:26.917316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:26.917013 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" event={"ID":"a2e14a0a-db97-451a-b942-171d5d643415","Type":"ContainerStarted","Data":"a8a52d7ca19ed067a89b08355fec62b4d0aa5ee149af0e38dcd6f203fbbb41ee"} Oct 31 03:37:27.043621 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:27.043572 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podStartSLOduration=2.043558326 podStartE2EDuration="2.043558326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:27.04214221 +0000 UTC m=+1080.360884838" watchObservedRunningTime="2026-10-31 03:37:27.043558326 +0000 UTC m=+1080.362300951" Oct 31 03:37:27.495252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:27.495157 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:27.495449 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:27.495305 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:37:27.495449 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:27.495358 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 03:37:27.562254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:27.561681 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:37:27.562254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:27.561762 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 03:37:28.493658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:28.493610 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:37:28.494319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:28.494281 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 03:37:28.573538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:28.573507 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:37:28.573774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:28.573751 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 03:37:29.495612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:29.495576 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:37:29.496193 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:29.496163 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 03:37:29.562774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:29.562745 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:37:29.563009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:29.562984 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 03:37:30.493641 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:30.493606 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:37:30.493940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:30.493914 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 03:37:30.564252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:30.563649 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:37:30.564252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:30.563703 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 03:37:31.492581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:31.492542 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:37:31.492779 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:31.492677 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 03:37:31.891528 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:31.891497 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:31.892706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:31.892670 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:31.893464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:31.893444 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:37:32.840871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:32.840847 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:32.841500 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:32.841470 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:32.842427 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:37:32.842403 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:37:49.167188 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:37:49.167143 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:38:06.706808 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.706733 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp"] Oct 31 03:38:06.711736 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.711720 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.815899 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.815871 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"distinct-2-w4-096812-kserve-self-signed-certs\"" Oct 31 03:38:06.817000 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.816966 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-home\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.817154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.817009 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tls-certs\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.817154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.817136 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tmp-dir\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.817297 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.817184 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnr4h\" (UniqueName: \"kubernetes.io/projected/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-kube-api-access-nnr4h\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.817366 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.817348 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-model-cache\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.817420 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.817386 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-dshm\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.820797 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.820778 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp"] Oct 31 03:38:06.918873 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.918825 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tmp-dir\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919067 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.918901 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnr4h\" (UniqueName: \"kubernetes.io/projected/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-kube-api-access-nnr4h\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919067 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.918993 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-model-cache\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919067 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.919025 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-dshm\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.919124 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-home\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.919157 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tls-certs\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919456 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.919431 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tmp-dir\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.919543 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-home\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.919872 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.919844 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-model-cache\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.921757 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.921735 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-dshm\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.922023 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.922007 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tls-certs\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:06.980394 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:06.980333 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnr4h\" (UniqueName: \"kubernetes.io/projected/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-kube-api-access-nnr4h\") pod \"distinct-2-w4-096812-kserve-f4f678fc6-pqfpp\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:07.024262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:07.024218 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:07.479381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:07.479158 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp"] Oct 31 03:38:07.482457 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:38:07.482427 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2012f9a6_651d_4c14_87ca_e0ec57a5f16d.slice/crio-6d9a8b85b51399da46dcbce3925f5f1b614efebf5aec7b16875bd59724a78aca WatchSource:0}: Error finding container 6d9a8b85b51399da46dcbce3925f5f1b614efebf5aec7b16875bd59724a78aca: Status 404 returned error can't find the container with id 6d9a8b85b51399da46dcbce3925f5f1b614efebf5aec7b16875bd59724a78aca Oct 31 03:38:08.234030 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:08.233997 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" event={"ID":"2012f9a6-651d-4c14-87ca-e0ec57a5f16d","Type":"ContainerStarted","Data":"4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2"} Oct 31 03:38:08.234030 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:08.234032 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" event={"ID":"2012f9a6-651d-4c14-87ca-e0ec57a5f16d","Type":"ContainerStarted","Data":"6d9a8b85b51399da46dcbce3925f5f1b614efebf5aec7b16875bd59724a78aca"} Oct 31 03:38:08.352762 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:08.352716 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podStartSLOduration=2.352703737 podStartE2EDuration="2.352703737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:08.350197643 +0000 UTC m=+1121.668940270" watchObservedRunningTime="2026-10-31 03:38:08.352703737 +0000 UTC m=+1121.671446362" Oct 31 03:38:09.024871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:09.024844 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:17.025111 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:17.025079 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:17.037492 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:17.037470 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:17.308938 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:17.308916 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:38:19.162716 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:38:19.162679 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:38:49.166456 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:38:49.166412 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:38:50.277613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.277580 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh"] Oct 31 03:38:50.281349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.281331 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.338424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.338391 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"openshift-service-ca.crt\"" Oct 31 03:38:50.348727 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.348704 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"distinct-2-w0-909365-kserve-self-signed-certs\"" Oct 31 03:38:50.355309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.355283 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fa25de-5c01-4e68-b69d-91699978b69f-tls-certs\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.355434 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.355332 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-tmp-dir\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.355494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.355430 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-dshm\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.355553 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.355519 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-home\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.355619 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.355598 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfw2s\" (UniqueName: \"kubernetes.io/projected/61fa25de-5c01-4e68-b69d-91699978b69f-kube-api-access-lfw2s\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.356313 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.355634 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-model-cache\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.388849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.388826 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"default-dockercfg-s5hcs\"" Oct 31 03:38:50.390997 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.390973 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh"] Oct 31 03:38:50.450700 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.450669 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"kube-root-ca.crt\"" Oct 31 03:38:50.473597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.473556 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfw2s\" (UniqueName: \"kubernetes.io/projected/61fa25de-5c01-4e68-b69d-91699978b69f-kube-api-access-lfw2s\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.480554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.479389 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-model-cache\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.480554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.479529 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fa25de-5c01-4e68-b69d-91699978b69f-tls-certs\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.480554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.479575 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-tmp-dir\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.480554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.479611 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-dshm\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.480554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.479680 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-home\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.480554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.480166 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-home\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.480554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.480461 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-model-cache\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.481307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.481146 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-tmp-dir\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.487781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.487639 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-dshm\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.491406 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.490021 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fa25de-5c01-4e68-b69d-91699978b69f-tls-certs\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.537474 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.537388 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfw2s\" (UniqueName: \"kubernetes.io/projected/61fa25de-5c01-4e68-b69d-91699978b69f-kube-api-access-lfw2s\") pod \"distinct-2-w0-909365-kserve-674b8ccd86-qpdgh\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:50.592922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:50.592893 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:51.116097 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:51.116069 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh"] Oct 31 03:38:51.120057 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:38:51.120024 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61fa25de_5c01_4e68_b69d_91699978b69f.slice/crio-d8777cf51cdee958871b2c064ba3d65893775241488e99c8fa2e1b4ce0a3a5bf WatchSource:0}: Error finding container d8777cf51cdee958871b2c064ba3d65893775241488e99c8fa2e1b4ce0a3a5bf: Status 404 returned error can't find the container with id d8777cf51cdee958871b2c064ba3d65893775241488e99c8fa2e1b4ce0a3a5bf Oct 31 03:38:51.508909 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:51.508823 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" event={"ID":"61fa25de-5c01-4e68-b69d-91699978b69f","Type":"ContainerStarted","Data":"9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a"} Oct 31 03:38:51.508909 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:51.508866 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" event={"ID":"61fa25de-5c01-4e68-b69d-91699978b69f","Type":"ContainerStarted","Data":"d8777cf51cdee958871b2c064ba3d65893775241488e99c8fa2e1b4ce0a3a5bf"} Oct 31 03:38:51.593665 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:51.593637 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:38:51.627943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:38:51.627899 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podStartSLOduration=1.627886995 podStartE2EDuration="1.627886995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:51.626546383 +0000 UTC m=+1164.945289007" watchObservedRunningTime="2026-10-31 03:38:51.627886995 +0000 UTC m=+1164.946629617" Oct 31 03:39:00.593084 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:00.593053 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:39:00.604896 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:00.604875 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:39:01.572296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:01.572268 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:39:19.167462 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:39:19.167420 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:39:25.850726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.850694 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz"] Oct 31 03:39:25.854688 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.854668 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.902681 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.902654 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.902812 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.902684 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.902812 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.902703 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe0bb92-7885-42de-bb32-9e3957822804-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.902812 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.902796 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.902921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.902855 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.902968 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.902949 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.903007 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.902997 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrhhn\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-kube-api-access-zrhhn\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.903059 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.903049 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe0bb92-7885-42de-bb32-9e3957822804-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.903105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.903094 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:25.953022 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:25.952997 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz"] Oct 31 03:39:26.003879 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.003854 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.003889 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.003919 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe0bb92-7885-42de-bb32-9e3957822804-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.003953 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004021 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004090 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004128 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhhn\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-kube-api-access-zrhhn\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004219 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe0bb92-7885-42de-bb32-9e3957822804-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004264 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004482 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004305 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004482 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004374 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004637 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004616 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.004940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.004921 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe0bb92-7885-42de-bb32-9e3957822804-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.005124 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.005108 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.006405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.006383 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.006638 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.006618 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe0bb92-7885-42de-bb32-9e3957822804-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.065333 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.065307 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.065422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.065380 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrhhn\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-kube-api-access-zrhhn\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.167878 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.167813 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:26.714727 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.714696 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz"] Oct 31 03:39:26.716819 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:39:26.716778 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe0bb92_7885_42de_bb32_9e3957822804.slice/crio-7c952fdbeee37648c73a132ce2fcb8ba9194565b5e5aadbe2ac6cbfb22815db5 WatchSource:0}: Error finding container 7c952fdbeee37648c73a132ce2fcb8ba9194565b5e5aadbe2ac6cbfb22815db5: Status 404 returned error can't find the container with id 7c952fdbeee37648c73a132ce2fcb8ba9194565b5e5aadbe2ac6cbfb22815db5 Oct 31 03:39:26.720192 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.720155 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:39:26.720302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.720261 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:39:26.720361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.720311 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:39:26.775830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:26.775804 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" event={"ID":"ffe0bb92-7885-42de-bb32-9e3957822804","Type":"ContainerStarted","Data":"7c952fdbeee37648c73a132ce2fcb8ba9194565b5e5aadbe2ac6cbfb22815db5"} Oct 31 03:39:27.137801 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:27.137764 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:39:27.143460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:27.143435 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:39:27.807572 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:27.807536 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" event={"ID":"ffe0bb92-7885-42de-bb32-9e3957822804","Type":"ContainerStarted","Data":"410ebc613723e652d6d568e4f7d9c34b7133c09c5b237141072577ceb4a0adc5"} Oct 31 03:39:27.930259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:27.920871 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b"] Oct 31 03:39:27.961527 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:27.961497 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.000278 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.000220 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b"] Oct 31 03:39:28.039251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.036283 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"distinct-2-w3-b98893-kserve-self-signed-certs\"" Oct 31 03:39:28.039251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.036738 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podStartSLOduration=3.036723116 podStartE2EDuration="3.036723116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:28.012664185 +0000 UTC m=+1201.331406813" watchObservedRunningTime="2026-10-31 03:39:28.036723116 +0000 UTC m=+1201.355465743" Oct 31 03:39:28.208325 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.208218 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:28.247431 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.209555 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-tmp-dir\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.248005 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.247981 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-dshm\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.248575 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.248551 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-home\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.248759 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.248735 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edb95a7-fa74-487f-ab8a-7658305c77ee-tls-certs\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.249191 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.249174 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-model-cache\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.249375 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.249349 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ctk\" (UniqueName: \"kubernetes.io/projected/6edb95a7-fa74-487f-ab8a-7658305c77ee-kube-api-access-l2ctk\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.250969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.250948 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:39:28.251168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.251129 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.420133 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-tmp-dir\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.420191 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-dshm\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.420326 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-home\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.420378 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edb95a7-fa74-487f-ab8a-7658305c77ee-tls-certs\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.420463 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-model-cache\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.420502 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ctk\" (UniqueName: \"kubernetes.io/projected/6edb95a7-fa74-487f-ab8a-7658305c77ee-kube-api-access-l2ctk\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.421533 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-tmp-dir\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.422198 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-model-cache\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.422852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.422477 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-home\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.448262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.433765 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-dshm\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.488278 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.486165 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ctk\" (UniqueName: \"kubernetes.io/projected/6edb95a7-fa74-487f-ab8a-7658305c77ee-kube-api-access-l2ctk\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.507253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.505459 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edb95a7-fa74-487f-ab8a-7658305c77ee-tls-certs\") pod \"distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.619254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.617458 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:28.676654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:28.676335 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.231194 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:29.231152 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.492607 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:29.465338 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:39:29.492607 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:29.465400 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 03:39:29.683254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:29.660443 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b"] Oct 31 03:39:29.954629 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:29.954526 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" event={"ID":"6edb95a7-fa74-487f-ab8a-7658305c77ee","Type":"ContainerStarted","Data":"dddb8877c18c8fff38629f3c075b30d38324c00f7add0e641691202bac73ebb4"} Oct 31 03:39:30.175382 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:30.174950 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:39:30.175382 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:30.175008 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 03:39:30.985199 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:30.985146 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" event={"ID":"6edb95a7-fa74-487f-ab8a-7658305c77ee","Type":"ContainerStarted","Data":"c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8"} Oct 31 03:39:31.107696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:31.107636 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podStartSLOduration=4.107617976 podStartE2EDuration="4.107617976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:31.103596741 +0000 UTC m=+1204.422339369" watchObservedRunningTime="2026-10-31 03:39:31.107617976 +0000 UTC m=+1204.426360604" Oct 31 03:39:31.172582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:31.172547 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:39:31.172837 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:31.172810 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 03:39:31.618136 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:31.618104 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:32.176253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:32.175598 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:39:32.176253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:32.175655 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 03:39:33.168580 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:33.168545 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:39:33.168910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:33.168880 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 03:39:34.168301 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:34.168264 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:39:34.168789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:34.168325 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 03:39:35.173368 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:35.173336 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:35.173776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:35.173729 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:35.174510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:35.174491 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 03:39:38.618436 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:38.618362 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:38.630352 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:38.630326 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:39.045006 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:39.044931 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:39:41.246119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.246083 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7"] Oct 31 03:39:41.250307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.250287 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331084 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331217 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331300 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331336 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331484 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331539 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331511 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331539 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331531 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331664 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331643 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kpl\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-kube-api-access-s5kpl\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.331746 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.331684 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.355329 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.355303 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7"] Oct 31 03:39:41.432652 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.432629 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.432770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.432664 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.432770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.432687 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.432770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.432766 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.432895 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.432789 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.432947 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.432924 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.432990 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kpl\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-kube-api-access-s5kpl\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433074 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.433028 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.433108 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433178 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.433145 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.433221 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.433323 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433462 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.433444 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.433532 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.433510 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.435392 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.435371 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.435485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.435443 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.494032 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.494012 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.494194 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.494177 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kpl\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-kube-api-access-s5kpl\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.563103 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.563083 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:41.991974 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.991950 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7"] Oct 31 03:39:41.993800 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:39:41.993777 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2e050d_524e_4844_ba8c_9e4955f9ff7c.slice/crio-dd32636a74d0f5eb227b58bc16f22a4fbbe2876b216e5bbc2d2128eb87d5e754 WatchSource:0}: Error finding container dd32636a74d0f5eb227b58bc16f22a4fbbe2876b216e5bbc2d2128eb87d5e754: Status 404 returned error can't find the container with id dd32636a74d0f5eb227b58bc16f22a4fbbe2876b216e5bbc2d2128eb87d5e754 Oct 31 03:39:41.996182 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.996155 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:39:41.996259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.996211 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:39:41.996309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:41.996283 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:39:42.053460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:42.053424 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" event={"ID":"2b2e050d-524e-4844-ba8c-9e4955f9ff7c","Type":"ContainerStarted","Data":"dd32636a74d0f5eb227b58bc16f22a4fbbe2876b216e5bbc2d2128eb87d5e754"} Oct 31 03:39:43.085325 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:43.083592 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" event={"ID":"2b2e050d-524e-4844-ba8c-9e4955f9ff7c","Type":"ContainerStarted","Data":"ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50"} Oct 31 03:39:43.284330 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:43.261204 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podStartSLOduration=2.261183689 podStartE2EDuration="2.261183689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:43.260372958 +0000 UTC m=+1216.579115586" watchObservedRunningTime="2026-10-31 03:39:43.261183689 +0000 UTC m=+1216.579926316" Oct 31 03:39:43.565251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:43.564769 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:43.566249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:43.565537 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:39:43.566249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:43.565586 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:39:44.568255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:44.566253 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:39:44.568255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:44.566315 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:39:45.567857 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:45.567823 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:39:45.568565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:45.568533 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:39:46.568334 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:46.568294 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:39:46.568621 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:46.568586 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:39:47.564772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:47.564739 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:39:47.565460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:47.565427 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:39:48.563751 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:48.563717 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:39:48.564107 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:48.563775 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:39:49.164752 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:39:49.164715 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:39:49.567667 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:49.567635 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:49.568136 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:49.568012 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:39:49.568628 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:39:49.568612 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:40:10.946250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:10.946205 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs"] Oct 31 03:40:10.951329 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:10.951307 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.057043 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.057010 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs"] Oct 31 03:40:11.060817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.060788 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.060958 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.060850 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.060958 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.060942 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvztj\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-kube-api-access-vvztj\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.061086 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.061004 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.061143 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.061102 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.061143 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.061131 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.061303 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.061273 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.061448 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.061335 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.061448 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.061425 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.162919 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.162889 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.162928 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.162948 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvztj\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-kube-api-access-vvztj\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.162996 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163025 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163051 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163185 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163271 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163315 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163354 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163389 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163466 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.163753 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.163714 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.164461 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.164434 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.165676 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.165651 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.165750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.165702 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.226275 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.226215 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvztj\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-kube-api-access-vvztj\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.226275 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.226221 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.266437 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.266416 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:11.694211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.694185 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs"] Oct 31 03:40:11.696525 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:40:11.696499 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abf3a24_37a8_422d_8b1b_d14a65fb903a.slice/crio-90ccd5c0151679870240cfd5a058b6eebb9b24eb0e28fca78fcfd4af98d9ea4c WatchSource:0}: Error finding container 90ccd5c0151679870240cfd5a058b6eebb9b24eb0e28fca78fcfd4af98d9ea4c: Status 404 returned error can't find the container with id 90ccd5c0151679870240cfd5a058b6eebb9b24eb0e28fca78fcfd4af98d9ea4c Oct 31 03:40:11.698678 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.698649 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:11.698765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.698726 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:11.698765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:11.698752 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:12.305463 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:12.305427 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" event={"ID":"0abf3a24-37a8-422d-8b1b-d14a65fb903a","Type":"ContainerStarted","Data":"0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326"} Oct 31 03:40:12.305958 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:12.305485 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" event={"ID":"0abf3a24-37a8-422d-8b1b-d14a65fb903a","Type":"ContainerStarted","Data":"90ccd5c0151679870240cfd5a058b6eebb9b24eb0e28fca78fcfd4af98d9ea4c"} Oct 31 03:40:12.433501 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:12.433450 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podStartSLOduration=2.43343594 podStartE2EDuration="2.43343594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:12.431183231 +0000 UTC m=+1245.749925860" watchObservedRunningTime="2026-10-31 03:40:12.43343594 +0000 UTC m=+1245.752178606" Oct 31 03:40:13.273253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:13.271178 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:13.273253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:13.271368 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:40:13.273253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:13.271415 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:40:14.279658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:14.279619 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:40:14.280153 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:14.279676 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:40:15.273464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:15.269567 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:40:15.273464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:15.269627 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:40:16.268254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:16.267938 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:40:16.268254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:16.268007 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:40:17.272255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:17.269579 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:40:17.272255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:17.269642 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:40:18.267488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:18.267450 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:40:18.267664 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:18.267505 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:40:19.164926 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:40:19.164887 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:40:19.271565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:19.271525 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:19.272014 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:19.271996 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:19.272704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:19.272683 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:40:26.376107 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.376076 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw"] Oct 31 03:40:26.380193 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.380166 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463103 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463066 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463272 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463195 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463272 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463269 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463387 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463285 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463445 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463432 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463465 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463536 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463491 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2z4x\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-kube-api-access-n2z4x\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463576 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463555 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.463620 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.463610 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.470949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.470914 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw"] Oct 31 03:40:26.564459 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564424 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564459 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564468 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564535 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564574 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564639 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2z4x\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-kube-api-access-n2z4x\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564670 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564714 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564793 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564867 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.564904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.564882 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.565186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.565168 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.565319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.565296 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.565566 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.565542 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.566334 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.566309 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.567643 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.567619 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.567762 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.567740 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.625787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.625763 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2z4x\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-kube-api-access-n2z4x\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.625787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.625775 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:26.695027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:26.694957 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:27.127515 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.127490 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw"] Oct 31 03:40:27.128878 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:40:27.128850 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4f5807_2d32_4c8d_91af_31e866dbb6ed.slice/crio-43f1633f54643438fdeb69396951d1bc86e4bd20c2327def5b9723999990a7f7 WatchSource:0}: Error finding container 43f1633f54643438fdeb69396951d1bc86e4bd20c2327def5b9723999990a7f7: Status 404 returned error can't find the container with id 43f1633f54643438fdeb69396951d1bc86e4bd20c2327def5b9723999990a7f7 Oct 31 03:40:27.131380 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.131336 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:27.131473 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.131409 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:27.131473 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.131442 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:27.396741 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.396648 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" event={"ID":"0e4f5807-2d32-4c8d-91af-31e866dbb6ed","Type":"ContainerStarted","Data":"f74c3d0e4ea9e51dc1235c94d4325d14589279148590235fcb1fb6d1a5d97a78"} Oct 31 03:40:27.396741 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.396709 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" event={"ID":"0e4f5807-2d32-4c8d-91af-31e866dbb6ed","Type":"ContainerStarted","Data":"43f1633f54643438fdeb69396951d1bc86e4bd20c2327def5b9723999990a7f7"} Oct 31 03:40:27.518759 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.518707 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podStartSLOduration=1.518692594 podStartE2EDuration="1.518692594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:27.517141195 +0000 UTC m=+1260.835883822" watchObservedRunningTime="2026-10-31 03:40:27.518692594 +0000 UTC m=+1260.837435220" Oct 31 03:40:27.695530 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:27.695449 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:28.697247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:28.697195 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:28.697746 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:28.697261 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:29.695452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:29.695411 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:29.695650 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:29.695469 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:30.696204 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:30.696164 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:30.696713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:30.696242 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:31.699253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:31.697865 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:31.699253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:31.697927 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:32.700261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:32.699335 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:32.700261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:32.699449 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:33.309255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.305247 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.333252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.328323 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.579440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.576442 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.677751 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.677706 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.711012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.710355 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:33.711012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.710419 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:33.774780 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.774322 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 03:40:33.839072 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.837307 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.844336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:33.843294 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 03:40:34.049369 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.049312 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.117565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.116501 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.337906 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.337863 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:40:34.358254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.338805 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:40:34.583265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.578414 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:40:34.583265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.579082 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:40:34.698258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.687278 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.703297 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.701946 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:34.703297 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.702013 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:34.708430 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.708341 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.815253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.811176 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.894780 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.894678 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 03:40:34.938444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:34.938396 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.072992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.072950 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.150851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.117456 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:40:35.324671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.324639 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:40:35.324922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.324907 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:40:35.325070 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.325058 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:40:35.325185 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.325175 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:40:35.326068 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.124403 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.359590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.338307 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.385255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.368784 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:40:35.385255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.358355 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.660305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.583361 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.706686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.706559 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:35.706968 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.706932 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:35.708106 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.708085 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded" Oct 31 03:40:35.740261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.710796 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:40:35.885292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.881580 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 03:40:35.912331 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:35.906187 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.283330 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.283135 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:40:36.288020 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.287983 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 03:40:36.400354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.400315 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:40:36.427264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.413049 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Oct 31 03:40:36.549960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.549870 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:40:36.568275 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.552496 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:40:36.594135 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.594068 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.764253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.751569 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:40:36.764253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.751628 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:40:36.796187 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.796148 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:40:36.805892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.805873 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:40:36.900551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.898320 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.907339 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.907306 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.927980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:36.927956 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:40:37.098536 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:37.097577 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:40:38.748405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:38.748357 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:38.750094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:38.748978 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:39.087340 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:39.087310 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:39.088273 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:39.088212 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:39.088919 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:39.088899 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:40:49.165275 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:40:49.165219 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:40:55.237483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.237449 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb"] Oct 31 03:40:55.241462 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.241446 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.335651 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.335616 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb"] Oct 31 03:40:55.399014 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.398988 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399179 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399115 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399275 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399181 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399345 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399293 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399400 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399431 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399460 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399550 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399539 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.399610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.399569 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkgf5\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-kube-api-access-bkgf5\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500152 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500196 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500217 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500285 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500302 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500328 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500353 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500390 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkgf5\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-kube-api-access-bkgf5\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500429 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500808 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500596 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500808 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500781 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.500808 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.500792 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.501140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.501123 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.501216 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.501196 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.502694 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.502676 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.503024 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.503003 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.561354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.561328 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.561438 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.561416 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkgf5\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-kube-api-access-bkgf5\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.855203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.855164 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:55.926306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.922182 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc"] Oct 31 03:40:55.930212 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.930187 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:55.986743 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:55.986710 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc"] Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005511 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005553 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005586 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005678 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbsb\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-kube-api-access-8xbsb\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005708 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11cf56-0806-4031-87a6-f5facaf9859b-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005768 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005835 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005866 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.006992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.005890 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107397 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107425 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107446 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107669 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107493 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbsb\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-kube-api-access-8xbsb\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107669 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107511 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11cf56-0806-4031-87a6-f5facaf9859b-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107669 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107553 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107669 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107596 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107782 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107816 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107924 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107826 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.107924 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.107911 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.108125 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.108101 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.108215 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.108132 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.108368 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.108341 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11cf56-0806-4031-87a6-f5facaf9859b-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.110284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.110267 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.110440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.110415 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.168412 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.168385 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.168807 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.168791 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbsb\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-kube-api-access-8xbsb\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.248152 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.248124 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:56.286630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.286603 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb"] Oct 31 03:40:56.288498 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:40:56.288191 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e73b68_b1ec_41c2_81bc_160ad40bce81.slice/crio-a2fa3c522fe126c3b2b808eea112faaf09ed69fb73b3ecadb339f4261237df92 WatchSource:0}: Error finding container a2fa3c522fe126c3b2b808eea112faaf09ed69fb73b3ecadb339f4261237df92: Status 404 returned error can't find the container with id a2fa3c522fe126c3b2b808eea112faaf09ed69fb73b3ecadb339f4261237df92 Oct 31 03:40:56.291119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.291080 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:56.291218 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.291162 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:56.291218 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.291210 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:56.311531 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.311510 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6"] Oct 31 03:40:56.317190 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.317170 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.378186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.378163 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6"] Oct 31 03:40:56.416268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.414590 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.416268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.416165 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.416268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.416267 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6878af03-5f4c-467c-bbb5-01fe97ade244-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.416512 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.416403 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnqd\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-kube-api-access-jvnqd\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.416570 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.416549 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.416993 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.416951 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.417347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.417270 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.417582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.417564 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.417840 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.417819 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.519393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.519350 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.519591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.519438 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.519787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.519755 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.519990 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.519972 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.520079 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.520065 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.520136 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.520092 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6878af03-5f4c-467c-bbb5-01fe97ade244-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.520189 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.520116 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnqd\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-kube-api-access-jvnqd\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.520570 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.520551 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.521000 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.520975 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.521276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.520278 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.521717 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.521691 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.521878 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.521863 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.522172 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.521281 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6878af03-5f4c-467c-bbb5-01fe97ade244-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.522488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.522468 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.522846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.522829 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.525677 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.525645 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.581666 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.581635 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnqd\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-kube-api-access-jvnqd\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.582004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.581987 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.638097 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.638057 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc"] Oct 31 03:40:56.640262 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:40:56.640199 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce11cf56_0806_4031_87a6_f5facaf9859b.slice/crio-dbf80d2eb36bf06006a5be3d94ea9b1db3b2647f073c61f65683ef7ef6fb702d WatchSource:0}: Error finding container dbf80d2eb36bf06006a5be3d94ea9b1db3b2647f073c61f65683ef7ef6fb702d: Status 404 returned error can't find the container with id dbf80d2eb36bf06006a5be3d94ea9b1db3b2647f073c61f65683ef7ef6fb702d Oct 31 03:40:56.643081 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.643045 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:56.643203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.643122 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:56.643203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.643166 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:56.670131 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.670106 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:56.953411 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.952964 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" event={"ID":"f5e73b68-b1ec-41c2-81bc-160ad40bce81","Type":"ContainerStarted","Data":"809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8"} Oct 31 03:40:56.953694 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.953673 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" event={"ID":"f5e73b68-b1ec-41c2-81bc-160ad40bce81","Type":"ContainerStarted","Data":"a2fa3c522fe126c3b2b808eea112faaf09ed69fb73b3ecadb339f4261237df92"} Oct 31 03:40:56.955665 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.955633 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" event={"ID":"ce11cf56-0806-4031-87a6-f5facaf9859b","Type":"ContainerStarted","Data":"31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668"} Oct 31 03:40:56.957131 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:56.957109 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" event={"ID":"ce11cf56-0806-4031-87a6-f5facaf9859b","Type":"ContainerStarted","Data":"dbf80d2eb36bf06006a5be3d94ea9b1db3b2647f073c61f65683ef7ef6fb702d"} Oct 31 03:40:57.090263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.083066 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6"] Oct 31 03:40:57.090263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.083752 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podStartSLOduration=2.083733632 podStartE2EDuration="2.083733632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:57.083560087 +0000 UTC m=+1290.402302714" watchObservedRunningTime="2026-10-31 03:40:57.083733632 +0000 UTC m=+1290.402476317" Oct 31 03:40:57.103886 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.103850 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:57.104015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.103921 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:57.104015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.103946 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:40:57.206198 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.206101 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podStartSLOduration=2.206082256 podStartE2EDuration="2.206082256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:57.203718939 +0000 UTC m=+1290.522461590" watchObservedRunningTime="2026-10-31 03:40:57.206082256 +0000 UTC m=+1290.524824882" Oct 31 03:40:57.249469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.248287 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:40:57.250205 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.250098 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:40:57.250205 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.250147 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 03:40:57.870252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.862564 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:40:57.870252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.868339 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:40:57.870252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:57.868406 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 03:40:58.001086 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.001049 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" event={"ID":"6878af03-5f4c-467c-bbb5-01fe97ade244","Type":"ContainerStarted","Data":"e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6"} Oct 31 03:40:58.005392 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.005364 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" event={"ID":"6878af03-5f4c-467c-bbb5-01fe97ade244","Type":"ContainerStarted","Data":"3b1fb6c8d9bd46a0d9c0869ba9512ae0c8d98b026d59c96194f192a3e781033f"} Oct 31 03:40:58.142597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.142473 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podStartSLOduration=2.142453633 podStartE2EDuration="2.142453633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:58.140724167 +0000 UTC m=+1291.459466796" watchObservedRunningTime="2026-10-31 03:40:58.142453633 +0000 UTC m=+1291.461196261" Oct 31 03:40:58.254671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.254633 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:40:58.255310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.255214 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 03:40:58.677484 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.677396 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:40:58.681656 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.681331 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:40:58.681656 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.681387 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:40:58.857726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.857643 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:40:58.891501 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:58.891352 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 03:40:59.248981 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:59.248947 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:40:59.249187 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:59.249007 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 03:40:59.686250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:59.685633 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:40:59.686250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:59.685693 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:40:59.863775 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:59.863742 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:40:59.864076 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:40:59.864032 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 03:41:00.389280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:00.386746 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:41:00.389280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:00.386819 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 03:41:00.782336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:00.774349 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:41:00.782336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:00.774410 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:41:00.863755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:00.863636 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:41:00.863755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:00.863694 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 03:41:01.286252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:01.286199 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:41:01.286542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:01.286507 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 03:41:01.706254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:01.672740 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:41:01.706254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:01.672842 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:41:01.975882 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:01.970872 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.001399 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.001348 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 03:41:02.328350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.303639 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.355934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.355893 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 03:41:02.429256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.426352 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.682307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.674345 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.682307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.674404 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:41:02.876309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.874201 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.876309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.874277 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 03:41:02.917254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:02.917202 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.258253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:03.254247 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:41:03.258253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:03.254317 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 03:41:03.681252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:03.673327 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:41:03.681252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:03.673430 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:41:03.858257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:03.855754 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:41:03.858257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:03.855813 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 03:41:04.694913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:04.694878 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:41:04.695499 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:04.695469 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:41:04.987342 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:04.982502 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:41:04.987342 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:04.983972 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:41:05.009814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:05.009547 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:41:05.010362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:05.010292 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:41:05.011818 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:05.010730 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:41:05.013658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:05.013566 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:41:05.864272 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:05.864197 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:41:05.866763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:05.866738 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:41:05.868538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:05.868517 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:41:10.427255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.418763 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb"] Oct 31 03:41:10.471614 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.471579 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.506662 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.506629 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb"] Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.632799 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8115d9-3924-42f4-96ec-29ecfd711637-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.632862 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.632894 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4gd\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-kube-api-access-qb4gd\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.632947 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.632985 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.633011 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.633055 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.633151 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.633263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.633272 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.800745 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.800647 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.801001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.800966 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.801300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.801280 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.807456 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.807430 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.807776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.807756 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.826032 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.826002 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8115d9-3924-42f4-96ec-29ecfd711637-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.826318 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.826288 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.826498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.826478 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4gd\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-kube-api-access-qb4gd\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.826702 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.826684 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.854326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.811981 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.854326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.811775 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.854326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.827912 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8115d9-3924-42f4-96ec-29ecfd711637-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.854326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.812193 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.854326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.828171 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.941270 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.931935 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.941270 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.932138 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:10.942138 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:10.942053 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:11.036484 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:11.036449 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4gd\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-kube-api-access-qb4gd\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:11.120524 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:11.120488 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:11.375051 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:11.374440 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:11.683295 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:11.682737 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:11.935350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:11.928615 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:11.944387 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:11.936848 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.275447 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.275350 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.356259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.355307 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.356259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.326320 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.381253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.375477 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.381253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.376098 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:41:12.381253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.376425 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:41:12.483262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.472351 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.627253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.602303 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:41:12.788249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.787183 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb"] Oct 31 03:41:12.975252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:12.971660 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 03:41:13.066704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.064941 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:13.073747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.073711 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:13.073988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.073968 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:13.315382 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.315331 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.406044 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.373396 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.418272 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.380911 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.418272 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.380945 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:13.427265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.422779 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:41:13.427265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.422840 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:41:13.427265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.422878 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:41:13.427265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.422895 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" event={"ID":"cc8115d9-3924-42f4-96ec-29ecfd711637","Type":"ContainerStarted","Data":"92b93be64a104e43a8d2d947a4f6e4d2cb329158e74053523897e46431bea087"} Oct 31 03:41:13.505415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.504286 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:13.506946 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.505685 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:41:13.770565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.770327 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.886255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.770476 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.901610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.901583 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:41:14.015253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:13.996344 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.215704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.037724 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:41:14.215704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.039021 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.478153 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.477832 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.505541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.505495 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:41:14.534251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.513568 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.571267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.566294 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.896976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.896931 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.904349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.904315 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:41:14.937767 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:14.937653 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 03:41:15.151211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.151167 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 03:41:15.283887 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.283793 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.287799 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.287766 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.516426 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.516379 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.517019 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.516892 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.623253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.621061 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.623253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.622916 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.942404 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.942292 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.992265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:41:15.981510 4934 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.032s" Oct 31 03:41:15.992265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.981595 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:41:15.992265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.981709 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:41:15.992265 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.981744 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:41:15.995676 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.993616 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" event={"ID":"cc8115d9-3924-42f4-96ec-29ecfd711637","Type":"ContainerStarted","Data":"bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f"} Oct 31 03:41:15.999119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:15.999091 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 03:41:16.046274 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.046245 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:41:16.254546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.254465 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:16.255540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.255515 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:16.256689 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.256658 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:16.315922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.298483 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:16.315922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.298795 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:16.315922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.299111 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:41:16.334593 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.334529 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:41:16.336690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.336552 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:41:16.336690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.336604 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:16.336690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.336640 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.377869 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.376547 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:41:16.390642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.389841 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.517934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.517826 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:41:16.621647 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.621616 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:41:16.653257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.627296 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:16.653257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.628038 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:41:16.653257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:16.628115 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.126622 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.126586 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:17.127251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.127203 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:17.184164 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.184101 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podStartSLOduration=7.184083902 podStartE2EDuration="7.184083902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:17.033777243 +0000 UTC m=+1310.352519870" watchObservedRunningTime="2026-10-31 03:41:17.184083902 +0000 UTC m=+1310.502826527" Oct 31 03:41:17.341278 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.203768 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:41:17.409681 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.409539 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.650494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.630396 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.652037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.651988 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:41:17.730334 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.729859 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:41:17.744680 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:17.744660 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:41:18.122858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:18.122828 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:18.123020 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:18.122883 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:19.121524 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:19.121486 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:19.121885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:19.121549 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:19.196089 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:41:19.196043 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:41:20.122157 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:20.122125 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:20.122682 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:20.122176 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:21.121916 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:21.121878 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:21.122200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:21.121936 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:22.123133 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:22.123098 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:22.123760 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:22.123729 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:23.121448 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:23.121410 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:23.121624 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:23.121475 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:24.121927 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:24.121888 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:41:24.122697 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:24.122639 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:41:25.126749 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:25.126715 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:25.127246 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:25.127054 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:25.127741 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:25.127722 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:41:29.879447 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:29.879415 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp"] Oct 31 03:41:29.884768 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:29.884747 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040372 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-certs\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040438 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-istio-token\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040495 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-data\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040544 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-socket\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040569 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-credential-socket\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040598 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44288212-950a-4166-958a-79a131755ca5-istiod-ca-cert\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040621 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4x5\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-kube-api-access-tj4x5\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040665 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44288212-950a-4166-958a-79a131755ca5-istio-podinfo\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.041250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.040741 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-envoy\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.042081 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.041488 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-67b1ef6f-openshift-default-dockercfg-9jqd8\"" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154105 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-certs\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154171 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-istio-token\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154247 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-data\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154302 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-socket\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154325 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-credential-socket\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154351 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44288212-950a-4166-958a-79a131755ca5-istiod-ca-cert\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154373 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4x5\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-kube-api-access-tj4x5\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154414 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44288212-950a-4166-958a-79a131755ca5-istio-podinfo\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.156300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.154494 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-envoy\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.176026 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.173960 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-socket\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.176026 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.174222 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-certs\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.186666 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.186634 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-credential-socket\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.187653 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.187627 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44288212-950a-4166-958a-79a131755ca5-istiod-ca-cert\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.189548 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.189526 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-data\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.191800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.191773 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp"] Oct 31 03:41:30.205427 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.204132 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44288212-950a-4166-958a-79a131755ca5-istio-podinfo\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.219515 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.219452 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-envoy\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.349271 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.335781 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-istio-token\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.349271 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.336320 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4x5\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-kube-api-access-tj4x5\") pod \"e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:30.499147 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:30.499023 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:31.025256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.025210 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp"] Oct 31 03:41:31.026837 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:41:31.026810 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44288212_950a_4166_958a_79a131755ca5.slice/crio-b1a6a9bf99f4de8744273c662058dd5df121be7048e45c749c9d27ee9a53e0c0 WatchSource:0}: Error finding container b1a6a9bf99f4de8744273c662058dd5df121be7048e45c749c9d27ee9a53e0c0: Status 404 returned error can't find the container with id b1a6a9bf99f4de8744273c662058dd5df121be7048e45c749c9d27ee9a53e0c0 Oct 31 03:41:31.029581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.029542 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:31.029661 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.029627 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:31.029711 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.029697 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:31.426148 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.426115 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" event={"ID":"44288212-950a-4166-958a-79a131755ca5","Type":"ContainerStarted","Data":"88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f"} Oct 31 03:41:31.426367 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.426180 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" event={"ID":"44288212-950a-4166-958a-79a131755ca5","Type":"ContainerStarted","Data":"b1a6a9bf99f4de8744273c662058dd5df121be7048e45c749c9d27ee9a53e0c0"} Oct 31 03:41:31.500044 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.500018 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:31.599052 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:31.598984 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podStartSLOduration=2.5989716 podStartE2EDuration="2.5989716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:31.597073927 +0000 UTC m=+1324.915816555" watchObservedRunningTime="2026-10-31 03:41:31.5989716 +0000 UTC m=+1324.917714226" Oct 31 03:41:32.504602 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:32.504574 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:33.438444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:33.438412 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:33.439467 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:33.439444 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:41:41.417248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.413302 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w"] Oct 31 03:41:41.432344 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.431678 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w"] Oct 31 03:41:41.432344 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.431744 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611262 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611451 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611316 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133fb05a-c4d1-4d03-a051-61bf044287f3-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611492 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611472 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611530 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611499 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611567 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611660 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611643 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611669 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611685 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.611846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.611827 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbfgm\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-kube-api-access-wbfgm\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713401 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713444 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713478 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713512 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713532 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713549 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713618 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbfgm\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-kube-api-access-wbfgm\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713730 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.713824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713790 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133fb05a-c4d1-4d03-a051-61bf044287f3-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.714289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713905 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.714289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.713968 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.714289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.714036 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.714459 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.714337 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.714553 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.714532 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133fb05a-c4d1-4d03-a051-61bf044287f3-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.716560 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.716540 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.716703 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.716686 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.776459 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.776431 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbfgm\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-kube-api-access-wbfgm\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:41.782703 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:41.782677 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:42.050666 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:42.050593 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:42.493712 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:41:42.493674 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133fb05a_c4d1_4d03_a051_61bf044287f3.slice/crio-6c95aa0a64a8da9cf51d5eaf4f5d9e2269a140d03840e7317d8c196ba4f9be3b WatchSource:0}: Error finding container 6c95aa0a64a8da9cf51d5eaf4f5d9e2269a140d03840e7317d8c196ba4f9be3b: Status 404 returned error can't find the container with id 6c95aa0a64a8da9cf51d5eaf4f5d9e2269a140d03840e7317d8c196ba4f9be3b Oct 31 03:41:42.494069 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:42.493742 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w"] Oct 31 03:41:42.496890 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:42.496855 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:42.496992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:42.496935 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:42.496992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:42.496974 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:41:42.547576 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:42.547548 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" event={"ID":"133fb05a-c4d1-4d03-a051-61bf044287f3","Type":"ContainerStarted","Data":"6c95aa0a64a8da9cf51d5eaf4f5d9e2269a140d03840e7317d8c196ba4f9be3b"} Oct 31 03:41:43.592253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:43.591305 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" event={"ID":"133fb05a-c4d1-4d03-a051-61bf044287f3","Type":"ContainerStarted","Data":"36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68"} Oct 31 03:41:43.826412 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:43.826363 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podStartSLOduration=2.826347906 podStartE2EDuration="2.826347906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:43.773698641 +0000 UTC m=+1337.092441270" watchObservedRunningTime="2026-10-31 03:41:43.826347906 +0000 UTC m=+1337.145090532" Oct 31 03:41:44.051190 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:44.051119 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:44.052375 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:44.052352 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:44.052508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:44.052414 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:45.051169 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:45.051136 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:45.051605 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:45.051199 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:46.058267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:46.053342 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:46.066474 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:46.066436 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:47.063311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:47.061532 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:47.063311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:47.061597 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:48.053254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:48.052675 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:48.053254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:48.052738 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:48.537398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:48.537346 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:41:48.635477 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:48.634327 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:48.715256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:48.693984 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.051253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.045338 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.246255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.233351 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:49.246255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.233412 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:49.408255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.407658 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.412250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.408538 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.497789 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:41:49.497645 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:41:49.541528 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.541442 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.542363 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.542306 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:41:49.576487 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.576447 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.621859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.621812 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.941253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.936982 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.962749 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.962697 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.963633 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:49.963597 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:41:50.071355 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.071305 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.080278 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.071924 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:41:50.115382 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.115341 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:50.139153 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.139119 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:50.159541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.159492 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:50.353537 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.353498 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.411482 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.411442 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.423848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.411749 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.436301 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.424120 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:41:50.436301 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.424190 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:41:50.552884 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.552842 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:41:50.584771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.583721 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:41:50.612345 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.612202 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:41:50.944981 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.944428 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.978317 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:50.978271 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 03:41:51.101011 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.100969 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.101319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.101293 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:51.122859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.115909 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:41:51.145252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.123192 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:51.145252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.123258 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:51.145252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.123283 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.145252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.124331 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:41:51.232317 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.231022 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:51.232317 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.231083 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.234786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.234669 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:51.234786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.234727 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.496348 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.431734 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 03:41:51.678964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.450968 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:51.678964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.597681 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.678964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.656883 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:41:51.678964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.604366 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.698936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.626726 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.698936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.628710 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.698936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.689679 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.965254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.946193 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:51.965254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.946297 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.018255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:51.992361 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 03:41:52.051264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.026625 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:41:52.134253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.109633 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:52.222314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.176731 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.383979 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.383940 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:52.462509 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.462471 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:52.505547 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.505456 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:41:52.554276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.533185 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.719257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.654941 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 03:41:52.891746 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.740383 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.892341 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.740440 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:52.892765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:52.740480 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.121253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.119967 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:53.121253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.120021 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:53.162980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.162881 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.182308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.164313 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.182308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.164320 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.182308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.177358 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.261256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.256542 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:41:53.392253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.389542 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:41:53.693295 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.693253 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:41:53.727258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.721729 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:41:53.727258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.723096 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.889581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.887004 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.952078 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:53.951997 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:41:54.070255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:54.070189 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:41:54.070442 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:54.070264 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 03:41:54.190370 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:54.190327 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:54.193426 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:54.191430 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:54.256057 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:54.255974 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:41:54.296252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:54.289749 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:41:54.296252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:54.293243 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:41:55.103406 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:55.103358 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:55.104192 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:55.104160 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:41:55.105039 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:41:55.105021 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:42:01.142873 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.142836 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn"] Oct 31 03:42:01.148848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.148826 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.204277 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.202583 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-6de37a-openshift-default-dockercfg-cfl84\"" Oct 31 03:42:01.249986 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.249955 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istiod-ca-cert\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250011 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-socket\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250270 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250180 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-envoy\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250324 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-credential-socket\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250436 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250409 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-certs\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250584 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250568 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-podinfo\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250611 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-token\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250661 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250649 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qw8\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-kube-api-access-d4qw8\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.250693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.250682 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-data\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.255550 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.255526 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn"] Oct 31 03:42:01.373254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.352192 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-socket\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.373987 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.373953 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-envoy\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.387259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.374303 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-credential-socket\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.390381 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-certs\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.390517 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-podinfo\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.390559 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-token\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.390596 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qw8\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-kube-api-access-d4qw8\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.390627 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-data\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.390695 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istiod-ca-cert\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.391503 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istiod-ca-cert\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.388146 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-socket\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.399114 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.391750 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-certs\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.451079 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.413339 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-credential-socket\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.451079 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.414067 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-data\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.495266 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.489250 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-token\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.521250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.514466 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-podinfo\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.554155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.546701 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-envoy\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.561150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.556946 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qw8\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-kube-api-access-d4qw8\") pod \"e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:01.766212 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:01.766139 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:02.206323 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:42:02.206291 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98adbaed_6f0f_4eeb_8e5a_1cd9b8bd293a.slice/crio-dc8a961c285c626bb2aea4dd17c567600de57ae09d545cae91debbf604de7600 WatchSource:0}: Error finding container dc8a961c285c626bb2aea4dd17c567600de57ae09d545cae91debbf604de7600: Status 404 returned error can't find the container with id dc8a961c285c626bb2aea4dd17c567600de57ae09d545cae91debbf604de7600 Oct 31 03:42:02.206323 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.206294 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn"] Oct 31 03:42:02.208917 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.208888 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:42:02.208985 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.208950 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:42:02.208985 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.208979 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:42:02.647316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.647264 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" event={"ID":"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a","Type":"ContainerStarted","Data":"85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483"} Oct 31 03:42:02.647489 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.647381 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" event={"ID":"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a","Type":"ContainerStarted","Data":"dc8a961c285c626bb2aea4dd17c567600de57ae09d545cae91debbf604de7600"} Oct 31 03:42:02.766943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.766914 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:02.820139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:02.820084 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podStartSLOduration=1.820070997 podStartE2EDuration="1.820070997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:02.81879376 +0000 UTC m=+1356.137536440" watchObservedRunningTime="2026-10-31 03:42:02.820070997 +0000 UTC m=+1356.138813623" Oct 31 03:42:03.772582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:03.772551 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:04.663835 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:04.663804 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:04.665051 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:04.665023 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:42:08.763575 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.763532 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd"] Oct 31 03:42:08.768706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.768688 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.807119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.807089 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e64da-2b05-4b82-b806-f29e3b09c202-openshift-service-ca\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.807221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.807127 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.807221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.807150 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-metrics-certs\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.807346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.807326 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5bm\" (UniqueName: \"kubernetes.io/projected/691e64da-2b05-4b82-b806-f29e3b09c202-kube-api-access-lz5bm\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.874092 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.874061 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dafd9df5-serving-cert\"" Oct 31 03:42:08.877027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.877002 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd"] Oct 31 03:42:08.909083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.909055 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e64da-2b05-4b82-b806-f29e3b09c202-openshift-service-ca\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.909243 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.909091 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.909243 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.909120 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-metrics-certs\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.909361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.909222 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5bm\" (UniqueName: \"kubernetes.io/projected/691e64da-2b05-4b82-b806-f29e3b09c202-kube-api-access-lz5bm\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.909361 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:08.909274 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-serving-cert: secret "maas-api-e2e-ait-dafd9df5-serving-cert" not found Oct 31 03:42:08.909361 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:08.909355 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls podName:691e64da-2b05-4b82-b806-f29e3b09c202 nodeName:}" failed. No retries permitted until 2026-10-31 03:42:09.409334804 +0000 UTC m=+1362.728077421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls") pod "maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" (UID: "691e64da-2b05-4b82-b806-f29e3b09c202") : secret "maas-api-e2e-ait-dafd9df5-serving-cert" not found Oct 31 03:42:08.910579 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.910549 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e64da-2b05-4b82-b806-f29e3b09c202-openshift-service-ca\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.912294 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.912221 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-metrics-certs\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:08.973051 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:08.973023 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5bm\" (UniqueName: \"kubernetes.io/projected/691e64da-2b05-4b82-b806-f29e3b09c202-kube-api-access-lz5bm\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:09.493408 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:09.493371 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:09.611975 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:09.611944 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls\") pod \"maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:09.690510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:09.690476 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:10.117444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:10.117419 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd"] Oct 31 03:42:10.118610 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:42:10.118584 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691e64da_2b05_4b82_b806_f29e3b09c202.slice/crio-daa4b773520ec637b972a987f1176d494d7e1ee3b45e3605ed8f7331172e9485 WatchSource:0}: Error finding container daa4b773520ec637b972a987f1176d494d7e1ee3b45e3605ed8f7331172e9485: Status 404 returned error can't find the container with id daa4b773520ec637b972a987f1176d494d7e1ee3b45e3605ed8f7331172e9485 Oct 31 03:42:10.120032 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:10.120014 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:42:10.725312 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:10.725274 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" event={"ID":"691e64da-2b05-4b82-b806-f29e3b09c202","Type":"ContainerStarted","Data":"daa4b773520ec637b972a987f1176d494d7e1ee3b45e3605ed8f7331172e9485"} Oct 31 03:42:11.732924 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:11.732889 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" event={"ID":"691e64da-2b05-4b82-b806-f29e3b09c202","Type":"ContainerStarted","Data":"29707972a135e0e4b78d8f2e7ade61d43fc19905d95b5c44b0da96f10fbb5ff0"} Oct 31 03:42:11.733414 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:11.733046 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:11.852299 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:11.852253 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" podStartSLOduration=3.010531405 podStartE2EDuration="3.852239812s" totalImagesPullingTime="841.708407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:11.850456123 +0000 UTC m=+1365.169198750" watchObservedRunningTime="2026-10-31 03:42:11.852239812 +0000 UTC m=+1365.170982461" Oct 31 03:42:14.149680 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.149650 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz"] Oct 31 03:42:14.153822 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.153804 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.209398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.209367 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9e38c94c-serving-cert\"" Oct 31 03:42:14.259415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.259387 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz"] Oct 31 03:42:14.308050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.308020 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllq7\" (UniqueName: \"kubernetes.io/projected/56496842-1c5e-440f-9bd6-480e1a35fae3-kube-api-access-qllq7\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.308220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.308180 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56496842-1c5e-440f-9bd6-480e1a35fae3-openshift-service-ca\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.308331 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.308220 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.308485 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.308459 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-metrics-certs\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.414008 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.413872 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllq7\" (UniqueName: \"kubernetes.io/projected/56496842-1c5e-440f-9bd6-480e1a35fae3-kube-api-access-qllq7\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.415274 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.415158 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56496842-1c5e-440f-9bd6-480e1a35fae3-openshift-service-ca\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.422340 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.422317 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.422770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.422748 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-metrics-certs\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.423560 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:14.422587 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-serving-cert: secret "maas-api-e2e-derive-9e38c94c-serving-cert" not found Oct 31 03:42:14.423732 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:14.423721 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls podName:56496842-1c5e-440f-9bd6-480e1a35fae3 nodeName:}" failed. No retries permitted until 2026-10-31 03:42:14.923699149 +0000 UTC m=+1368.242441758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls") pod "maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" (UID: "56496842-1c5e-440f-9bd6-480e1a35fae3") : secret "maas-api-e2e-derive-9e38c94c-serving-cert" not found Oct 31 03:42:14.424056 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.422199 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56496842-1c5e-440f-9bd6-480e1a35fae3-openshift-service-ca\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.461651 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.458901 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-metrics-certs\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.498064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.482780 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllq7\" (UniqueName: \"kubernetes.io/projected/56496842-1c5e-440f-9bd6-480e1a35fae3-kube-api-access-qllq7\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.937409 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.937374 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:14.939876 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:14.939845 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls\") pod \"maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:15.066648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:15.066622 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:15.452477 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:15.452451 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz"] Oct 31 03:42:15.454481 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:42:15.454456 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56496842_1c5e_440f_9bd6_480e1a35fae3.slice/crio-a021c47a3291fd70665201a9b7eea844496c3dbf9f49db6fd66ad95198b91aab WatchSource:0}: Error finding container a021c47a3291fd70665201a9b7eea844496c3dbf9f49db6fd66ad95198b91aab: Status 404 returned error can't find the container with id a021c47a3291fd70665201a9b7eea844496c3dbf9f49db6fd66ad95198b91aab Oct 31 03:42:15.764600 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:15.764517 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" event={"ID":"56496842-1c5e-440f-9bd6-480e1a35fae3","Type":"ContainerStarted","Data":"a021c47a3291fd70665201a9b7eea844496c3dbf9f49db6fd66ad95198b91aab"} Oct 31 03:42:16.775566 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:16.775535 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" event={"ID":"56496842-1c5e-440f-9bd6-480e1a35fae3","Type":"ContainerStarted","Data":"6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678"} Oct 31 03:42:16.776050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:16.775756 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:16.896368 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:16.896319 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" podStartSLOduration=2.083715836 podStartE2EDuration="2.896306468s" totalImagesPullingTime="812.590632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:16.892497255 +0000 UTC m=+1370.211239894" watchObservedRunningTime="2026-10-31 03:42:16.896306468 +0000 UTC m=+1370.215049117" Oct 31 03:42:17.745212 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:17.745188 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:42:19.163892 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:19.163860 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:42:22.788776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:22.788743 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:42:24.644435 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.644402 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx"] Oct 31 03:42:24.648787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.648772 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.701216 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.701190 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-67b1ef6f-serving-cert\"" Oct 31 03:42:24.714080 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.714050 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.714279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.714217 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-metrics-certs\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.714402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.714304 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8shh\" (UniqueName: \"kubernetes.io/projected/5b04003a-7ae4-42c5-bb33-246aea5a0a48-kube-api-access-s8shh\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.714402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.714367 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b04003a-7ae4-42c5-bb33-246aea5a0a48-openshift-service-ca\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.754448 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.754425 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx"] Oct 31 03:42:24.815429 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.815394 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-metrics-certs\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.815582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.815444 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8shh\" (UniqueName: \"kubernetes.io/projected/5b04003a-7ae4-42c5-bb33-246aea5a0a48-kube-api-access-s8shh\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.815582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.815485 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b04003a-7ae4-42c5-bb33-246aea5a0a48-openshift-service-ca\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.815689 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.815600 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.815746 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:24.815707 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-serving-cert: secret "maas-api-e2e-ait-67b1ef6f-serving-cert" not found Oct 31 03:42:24.815797 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:24.815773 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls podName:5b04003a-7ae4-42c5-bb33-246aea5a0a48 nodeName:}" failed. No retries permitted until 2026-10-31 03:42:25.315753709 +0000 UTC m=+1378.634496334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls") pod "maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" (UID: "5b04003a-7ae4-42c5-bb33-246aea5a0a48") : secret "maas-api-e2e-ait-67b1ef6f-serving-cert" not found Oct 31 03:42:24.816161 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.816139 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b04003a-7ae4-42c5-bb33-246aea5a0a48-openshift-service-ca\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.818063 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.818030 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-metrics-certs\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:24.877090 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:24.877066 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8shh\" (UniqueName: \"kubernetes.io/projected/5b04003a-7ae4-42c5-bb33-246aea5a0a48-kube-api-access-s8shh\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:25.326601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:25.326569 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:25.329187 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:25.329159 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls\") pod \"maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:25.571175 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:25.571139 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:25.949263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:25.949218 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx"] Oct 31 03:42:25.950144 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:42:25.950114 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b04003a_7ae4_42c5_bb33_246aea5a0a48.slice/crio-233f8028d194a6af7bd547a25f192c23b79327fb125b65ea3572bb4145284fd0 WatchSource:0}: Error finding container 233f8028d194a6af7bd547a25f192c23b79327fb125b65ea3572bb4145284fd0: Status 404 returned error can't find the container with id 233f8028d194a6af7bd547a25f192c23b79327fb125b65ea3572bb4145284fd0 Oct 31 03:42:26.916291 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:26.913639 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" event={"ID":"5b04003a-7ae4-42c5-bb33-246aea5a0a48","Type":"ContainerStarted","Data":"233f8028d194a6af7bd547a25f192c23b79327fb125b65ea3572bb4145284fd0"} Oct 31 03:42:27.922979 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:27.922943 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" event={"ID":"5b04003a-7ae4-42c5-bb33-246aea5a0a48","Type":"ContainerStarted","Data":"630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8"} Oct 31 03:42:27.923550 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:27.923531 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:28.045724 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:28.045677 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" podStartSLOduration=3.150904611 podStartE2EDuration="4.045664976s" totalImagesPullingTime="894.760365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:28.04243647 +0000 UTC m=+1381.361179098" watchObservedRunningTime="2026-10-31 03:42:28.045664976 +0000 UTC m=+1381.364407603" Oct 31 03:42:33.933988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:33.933959 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:42:38.224501 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:38.224399 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.571504 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:38.571458 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.685365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:38.685313 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.687261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:38.686789 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.748254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:38.744344 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.894410 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:38.894311 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.895380 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:38.894411 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.036438 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.036364 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.151686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.150498 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.183352 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.183309 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.236322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.236253 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.237270 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.237249 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:42:39.634587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.634542 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.636659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.636636 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:42:39.682300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.650904 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.700255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.699814 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.700255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.700219 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:42:39.746417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.745572 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.746417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.745983 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:42:39.921591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.921463 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.933078 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.933036 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.955066 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.955036 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:42:39.955349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:39.955333 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:42:40.039418 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.039374 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 03:42:40.040102 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.040076 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:42:40.171260 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.159308 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.171260 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.159698 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:42:40.230432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.185459 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:42:40.253261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.242344 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:42:40.253261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.242421 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.253261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.242511 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:42:40.284333 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.272433 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.284333 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.272827 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:42:40.476258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.476211 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:42:40.653255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.653021 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:42:40.700406 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.700361 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.755372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.754248 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.768134 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:40.768110 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:42:41.074306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.057431 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 03:42:41.074306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.057857 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.166251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.160451 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.278353 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.278307 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded" Oct 31 03:42:41.623396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.623347 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.670988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.670943 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.737928 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.737887 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.994005 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:41.993917 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:42:42.050521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:42.050492 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:42:42.067730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:42.067678 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.069498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:42.069477 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:42:42.070292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:42.070277 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:42:42.087132 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:42.087086 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.136365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:42.136336 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:42:42.138241 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:42:42.138206 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:42:49.164803 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:42:49.164758 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:43:09.541091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.541049 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr"] Oct 31 03:43:09.547294 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.547274 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.610373 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.610340 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr"] Oct 31 03:43:09.652651 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.652604 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scqhw\" (UniqueName: \"kubernetes.io/projected/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-kube-api-access-scqhw\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.652799 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.652768 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.652869 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.652850 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.653038 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.653020 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.660083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.653787 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-6de37a-serving-cert\"" Oct 31 03:43:09.754304 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.754268 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.754470 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.754403 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqhw\" (UniqueName: \"kubernetes.io/projected/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-kube-api-access-scqhw\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.754470 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:43:09.754417 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-serving-cert: secret "maas-api-e2e-shared-a-w5-6de37a-serving-cert" not found Oct 31 03:43:09.754614 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:43:09.754490 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls podName:ed1fd977-8cf4-4938-8e6d-b1c2fb61faae nodeName:}" failed. No retries permitted until 2026-10-31 03:43:10.25446949 +0000 UTC m=+1423.573212097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls") pod "maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" (UID: "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae") : secret "maas-api-e2e-shared-a-w5-6de37a-serving-cert" not found Oct 31 03:43:09.754614 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.754515 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.754614 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.754589 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.755488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.755464 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.757705 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.757684 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:09.841786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:09.840470 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqhw\" (UniqueName: \"kubernetes.io/projected/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-kube-api-access-scqhw\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:10.280890 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:10.280806 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:10.283286 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:10.283258 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:10.461793 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:10.461751 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:10.854806 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:10.854781 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr"] Oct 31 03:43:10.856162 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:43:10.856128 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1fd977_8cf4_4938_8e6d_b1c2fb61faae.slice/crio-3378c49606f484097fcb62193644a532fc8eced963ffbbd69bf6064de353b3ca WatchSource:0}: Error finding container 3378c49606f484097fcb62193644a532fc8eced963ffbbd69bf6064de353b3ca: Status 404 returned error can't find the container with id 3378c49606f484097fcb62193644a532fc8eced963ffbbd69bf6064de353b3ca Oct 31 03:43:11.230871 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:11.230788 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" event={"ID":"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae","Type":"ContainerStarted","Data":"3378c49606f484097fcb62193644a532fc8eced963ffbbd69bf6064de353b3ca"} Oct 31 03:43:12.239450 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:12.239414 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" event={"ID":"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae","Type":"ContainerStarted","Data":"2a17597e8612c3f2130578b8955bc8d3a14cdf8e4f8495d64e39d81223070864"} Oct 31 03:43:12.239882 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:12.239486 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:12.360032 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:12.359982 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" podStartSLOduration=2.439249608 podStartE2EDuration="3.359968527s" totalImagesPullingTime="920.718919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:12.35683606 +0000 UTC m=+1425.675578703" watchObservedRunningTime="2026-10-31 03:43:12.359968527 +0000 UTC m=+1425.678711187" Oct 31 03:43:18.251608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:18.251582 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 03:43:19.184759 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:43:19.184718 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:43:45.529686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:45.529652 4934 generic.go:356] "Generic (PLEG): container finished" podID="44288212-950a-4166-958a-79a131755ca5" containerID="88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f" exitCode=0 Oct 31 03:43:45.530027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:45.529724 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" event={"ID":"44288212-950a-4166-958a-79a131755ca5","Type":"ContainerDied","Data":"88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f"} Oct 31 03:43:45.530121 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:45.530108 4934 scope.go:122] "RemoveContainer" containerID="88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f" Oct 31 03:43:45.531707 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:45.531680 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:43:45.531780 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:45.531756 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:43:45.531817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:45.531788 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:43:46.562862 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:46.562824 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" event={"ID":"44288212-950a-4166-958a-79a131755ca5","Type":"ContainerStarted","Data":"a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e"} Oct 31 03:43:47.499889 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:47.499847 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:43:47.501393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:47.501364 4934 patch_prober.go:29] interesting pod/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:43:47.501534 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:47.501423 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:43:48.438529 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:48.438496 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:43:48.499351 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:48.499314 4934 patch_prober.go:29] interesting pod/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:43:48.499503 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:48.499375 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:43:49.199100 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:43:49.199060 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:43:49.499852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:49.499763 4934 patch_prober.go:29] interesting pod/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:43:49.499852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:49.499823 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:43:50.500352 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:50.500314 4934 patch_prober.go:29] interesting pod/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:43:50.500778 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:50.500412 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:43:51.500382 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:51.500348 4934 patch_prober.go:29] interesting pod/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:43:51.500813 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:51.500408 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:43:52.504795 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:52.504767 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:43:52.505987 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:43:52.505966 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:44:04.875424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:04.874791 4934 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:46380->10.132.0.68:15021: read: connection reset by peer" start-of-body= Oct 31 03:44:04.875424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:04.874844 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:46380->10.132.0.68:15021: read: connection reset by peer" Oct 31 03:44:05.743910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:05.743877 4934 generic.go:356] "Generic (PLEG): container finished" podID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerID="85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483" exitCode=0 Oct 31 03:44:05.744070 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:05.743930 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" event={"ID":"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a","Type":"ContainerDied","Data":"85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483"} Oct 31 03:44:05.744370 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:05.744354 4934 scope.go:122] "RemoveContainer" containerID="85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483" Oct 31 03:44:05.746135 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:05.746097 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:44:05.746251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:05.746194 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:44:05.746298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:05.746262 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:44:06.752720 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:06.752640 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" event={"ID":"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a","Type":"ContainerStarted","Data":"f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1"} Oct 31 03:44:06.778508 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:06.778468 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:44:06.778782 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:06.778631 4934 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:44:06.778918 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:06.778800 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 03:44:07.773251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:07.766558 4934 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:44:07.773251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:07.766623 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 03:44:08.767444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:08.767410 4934 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:44:08.767723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:08.767698 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 03:44:09.769551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:09.769513 4934 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:44:09.770141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:09.770110 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 03:44:10.793361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:10.787483 4934 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:44:10.793361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:10.787565 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 03:44:11.766629 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:11.766602 4934 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:44:11.766988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:11.766949 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 03:44:12.770596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:12.770566 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:44:12.771089 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:12.770954 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:44:12.771724 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:12.771696 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 03:44:19.164185 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:44:19.164150 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:44:20.209635 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:20.209595 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd"] Oct 31 03:44:20.228253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:20.210445 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" podUID="691e64da-2b05-4b82-b806-f29e3b09c202" containerName="maas-api" containerID="cri-o://29707972a135e0e4b78d8f2e7ade61d43fc19905d95b5c44b0da96f10fbb5ff0" gracePeriod=30 Oct 31 03:44:21.078907 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.078870 4934 generic.go:356] "Generic (PLEG): container finished" podID="691e64da-2b05-4b82-b806-f29e3b09c202" containerID="29707972a135e0e4b78d8f2e7ade61d43fc19905d95b5c44b0da96f10fbb5ff0" exitCode=0 Oct 31 03:44:21.090325 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.090282 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" event={"ID":"691e64da-2b05-4b82-b806-f29e3b09c202","Type":"ContainerDied","Data":"29707972a135e0e4b78d8f2e7ade61d43fc19905d95b5c44b0da96f10fbb5ff0"} Oct 31 03:44:21.246538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.246504 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:44:21.303722 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.303684 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-metrics-certs\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-metrics-certs\") pod \"691e64da-2b05-4b82-b806-f29e3b09c202\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " Oct 31 03:44:21.303949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.303869 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691e64da-2b05-4b82-b806-f29e3b09c202-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e64da-2b05-4b82-b806-f29e3b09c202-openshift-service-ca\") pod \"691e64da-2b05-4b82-b806-f29e3b09c202\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " Oct 31 03:44:21.304010 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.303985 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls\") pod \"691e64da-2b05-4b82-b806-f29e3b09c202\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " Oct 31 03:44:21.304063 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.304033 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691e64da-2b05-4b82-b806-f29e3b09c202-kube-api-access-lz5bm\" (UniqueName: \"kubernetes.io/projected/691e64da-2b05-4b82-b806-f29e3b09c202-kube-api-access-lz5bm\") pod \"691e64da-2b05-4b82-b806-f29e3b09c202\" (UID: \"691e64da-2b05-4b82-b806-f29e3b09c202\") " Oct 31 03:44:21.304759 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.304727 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691e64da-2b05-4b82-b806-f29e3b09c202-openshift-service-ca" pod "691e64da-2b05-4b82-b806-f29e3b09c202" (UID: "691e64da-2b05-4b82-b806-f29e3b09c202"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:21.308124 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.308098 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls" pod "691e64da-2b05-4b82-b806-f29e3b09c202" (UID: "691e64da-2b05-4b82-b806-f29e3b09c202"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:44:21.308304 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.308190 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691e64da-2b05-4b82-b806-f29e3b09c202-kube-api-access-lz5bm" pod "691e64da-2b05-4b82-b806-f29e3b09c202" (UID: "691e64da-2b05-4b82-b806-f29e3b09c202"). InnerVolumeSpecName "kube-api-access-lz5bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:21.402012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.401919 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-metrics-certs" pod "691e64da-2b05-4b82-b806-f29e3b09c202" (UID: "691e64da-2b05-4b82-b806-f29e3b09c202"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:44:21.410017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.409985 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.410243 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.410210 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz5bm\" (UniqueName: \"kubernetes.io/projected/691e64da-2b05-4b82-b806-f29e3b09c202-kube-api-access-lz5bm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.410346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.410333 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/691e64da-2b05-4b82-b806-f29e3b09c202-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.410420 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.410410 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e64da-2b05-4b82-b806-f29e3b09c202-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.449398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.449368 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz"] Oct 31 03:44:21.449595 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.449559 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" podUID="56496842-1c5e-440f-9bd6-480e1a35fae3" containerName="maas-api" containerID="cri-o://6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678" gracePeriod=30 Oct 31 03:44:21.886296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:21.886274 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:44:22.050306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.049728 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56496842-1c5e-440f-9bd6-480e1a35fae3-kube-api-access-qllq7\" (UniqueName: \"kubernetes.io/projected/56496842-1c5e-440f-9bd6-480e1a35fae3-kube-api-access-qllq7\") pod \"56496842-1c5e-440f-9bd6-480e1a35fae3\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " Oct 31 03:44:22.050306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.049780 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-metrics-certs\") pod \"56496842-1c5e-440f-9bd6-480e1a35fae3\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " Oct 31 03:44:22.050306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.049814 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls\") pod \"56496842-1c5e-440f-9bd6-480e1a35fae3\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " Oct 31 03:44:22.050306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.050096 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56496842-1c5e-440f-9bd6-480e1a35fae3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56496842-1c5e-440f-9bd6-480e1a35fae3-openshift-service-ca\") pod \"56496842-1c5e-440f-9bd6-480e1a35fae3\" (UID: \"56496842-1c5e-440f-9bd6-480e1a35fae3\") " Oct 31 03:44:22.059322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.059266 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56496842-1c5e-440f-9bd6-480e1a35fae3-kube-api-access-qllq7" pod "56496842-1c5e-440f-9bd6-480e1a35fae3" (UID: "56496842-1c5e-440f-9bd6-480e1a35fae3"). InnerVolumeSpecName "kube-api-access-qllq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:22.060079 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.060051 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56496842-1c5e-440f-9bd6-480e1a35fae3-openshift-service-ca" pod "56496842-1c5e-440f-9bd6-480e1a35fae3" (UID: "56496842-1c5e-440f-9bd6-480e1a35fae3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:22.100403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.100366 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-metrics-certs" pod "56496842-1c5e-440f-9bd6-480e1a35fae3" (UID: "56496842-1c5e-440f-9bd6-480e1a35fae3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:44:22.107864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.107830 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls" pod "56496842-1c5e-440f-9bd6-480e1a35fae3" (UID: "56496842-1c5e-440f-9bd6-480e1a35fae3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:44:22.131296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.129107 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" event={"ID":"691e64da-2b05-4b82-b806-f29e3b09c202","Type":"ContainerDied","Data":"daa4b773520ec637b972a987f1176d494d7e1ee3b45e3605ed8f7331172e9485"} Oct 31 03:44:22.131296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.129182 4934 scope.go:122] "RemoveContainer" containerID="29707972a135e0e4b78d8f2e7ade61d43fc19905d95b5c44b0da96f10fbb5ff0" Oct 31 03:44:22.131296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.129378 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd" Oct 31 03:44:22.142094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.142067 4934 generic.go:356] "Generic (PLEG): container finished" podID="56496842-1c5e-440f-9bd6-480e1a35fae3" containerID="6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678" exitCode=0 Oct 31 03:44:22.142783 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.142758 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" event={"ID":"56496842-1c5e-440f-9bd6-480e1a35fae3","Type":"ContainerDied","Data":"6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678"} Oct 31 03:44:22.143003 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.142967 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" event={"ID":"56496842-1c5e-440f-9bd6-480e1a35fae3","Type":"ContainerDied","Data":"a021c47a3291fd70665201a9b7eea844496c3dbf9f49db6fd66ad95198b91aab"} Oct 31 03:44:22.143263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.143221 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz" Oct 31 03:44:22.161631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.156279 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56496842-1c5e-440f-9bd6-480e1a35fae3-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:22.161631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.156309 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllq7\" (UniqueName: \"kubernetes.io/projected/56496842-1c5e-440f-9bd6-480e1a35fae3-kube-api-access-qllq7\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:22.161631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.156328 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:22.161631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.156342 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56496842-1c5e-440f-9bd6-480e1a35fae3-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:22.205586 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.205441 4934 scope.go:122] "RemoveContainer" containerID="6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678" Oct 31 03:44:22.220185 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.220156 4934 scope.go:122] "RemoveContainer" containerID="6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678" Oct 31 03:44:22.220581 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:44:22.220560 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678\": container with ID starting with 6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678 not found: ID does not exist" containerID="6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678" Oct 31 03:44:22.220747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.220713 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678"} err="failed to get container status \"6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678\": rpc error: code = NotFound desc = could not find container \"6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678\": container with ID starting with 6cbe42e7b32cff74dfc329ffcae276283baea66b69575f845248ab1405f01678 not found: ID does not exist" Oct 31 03:44:22.312554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.312524 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd"] Oct 31 03:44:22.318192 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.318167 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dafd9df5-6548f999b-qg2cd"] Oct 31 03:44:22.523254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.521423 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz"] Oct 31 03:44:22.529281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.527359 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9e38c94c-7bd4dd7c6-pwwvz"] Oct 31 03:44:22.857254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.851385 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56496842-1c5e-440f-9bd6-480e1a35fae3" path="/var/lib/kubelet/pods/56496842-1c5e-440f-9bd6-480e1a35fae3/volumes" Oct 31 03:44:22.867711 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:22.867663 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691e64da-2b05-4b82-b806-f29e3b09c202" path="/var/lib/kubelet/pods/691e64da-2b05-4b82-b806-f29e3b09c202/volumes" Oct 31 03:44:24.230250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.227984 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6"] Oct 31 03:44:24.252249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.244953 4934 cpu_manager.go:414] "containerMap: removing container" podUID="691e64da-2b05-4b82-b806-f29e3b09c202" containerName="maas-api" Oct 31 03:44:24.252249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.244979 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691e64da-2b05-4b82-b806-f29e3b09c202" containerName="maas-api" Oct 31 03:44:24.252249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.245034 4934 cpu_manager.go:414] "containerMap: removing container" podUID="56496842-1c5e-440f-9bd6-480e1a35fae3" containerName="maas-api" Oct 31 03:44:24.252249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.245043 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56496842-1c5e-440f-9bd6-480e1a35fae3" containerName="maas-api" Oct 31 03:44:24.252249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.245216 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="56496842-1c5e-440f-9bd6-480e1a35fae3" containerName="maas-api" Oct 31 03:44:24.252249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.245250 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="691e64da-2b05-4b82-b806-f29e3b09c202" containerName="maas-api" Oct 31 03:44:24.280264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.271564 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.330253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.312701 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6"] Oct 31 03:44:24.392734 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.392703 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bb362142-gw-openshift-default-dockercfg-llj8k\"" Oct 31 03:44:24.473256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.449163 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d149a2e7-c176-4787-8f1a-ba47853de5d2-istiod-ca-cert\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.526901 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.526810 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-podinfo\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.527172 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.527146 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-token\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.542279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.531164 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjx6q\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-kube-api-access-pjx6q\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.542574 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.542545 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-certs\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.542728 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.542713 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-data\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.542862 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.542848 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-credential-socket\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.542988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.542972 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-socket\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.543296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.543276 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-envoy\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.662764 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d149a2e7-c176-4787-8f1a-ba47853de5d2-istiod-ca-cert\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.662877 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-podinfo\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.662912 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-token\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.662975 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjx6q\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-kube-api-access-pjx6q\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.663046 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-certs\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.663109 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-data\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.663167 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-credential-socket\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.663212 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-socket\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.663856 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.663443 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-envoy\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.673405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.673375 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d149a2e7-c176-4787-8f1a-ba47853de5d2-istiod-ca-cert\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.673837 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.673818 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-certs\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.674240 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.674203 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-credential-socket\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.674637 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.674617 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-socket\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.683248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.675253 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-data\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.692779 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.684828 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-podinfo\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.737927 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.737892 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjx6q\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-kube-api-access-pjx6q\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.744252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.743598 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-envoy\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.751774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.751742 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-token\") pod \"e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:24.954587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:24.954555 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:25.849386 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:25.846315 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6"] Oct 31 03:44:25.973166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:25.973112 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:44:25.973620 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:25.973589 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:44:25.979747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:25.973791 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:44:26.259308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:26.259193 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" event={"ID":"d149a2e7-c176-4787-8f1a-ba47853de5d2","Type":"ContainerStarted","Data":"3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0"} Oct 31 03:44:26.259308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:26.259266 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" event={"ID":"d149a2e7-c176-4787-8f1a-ba47853de5d2","Type":"ContainerStarted","Data":"88ea422f8c0ee1c0a1914ac1af2b0167b3c0cc58968ceaab887273839f84f893"} Oct 31 03:44:26.384126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:26.384073 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podStartSLOduration=2.384058555 podStartE2EDuration="2.384058555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:26.380863186 +0000 UTC m=+1499.699605815" watchObservedRunningTime="2026-10-31 03:44:26.384058555 +0000 UTC m=+1499.702801197" Oct 31 03:44:26.960101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:26.960067 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:27.460935 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:27.460907 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:44:27.464042 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:27.464019 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:44:27.961087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:27.961054 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:28.202396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:28.202367 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx"] Oct 31 03:44:28.202935 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:28.202903 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" podUID="5b04003a-7ae4-42c5-bb33-246aea5a0a48" containerName="maas-api" containerID="cri-o://630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8" gracePeriod=30 Oct 31 03:44:28.316309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:28.312201 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:28.316309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:28.313161 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:44:28.989251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:28.984950 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:44:29.096285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.091592 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls\") pod \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " Oct 31 03:44:29.096285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.091747 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b04003a-7ae4-42c5-bb33-246aea5a0a48-kube-api-access-s8shh\" (UniqueName: \"kubernetes.io/projected/5b04003a-7ae4-42c5-bb33-246aea5a0a48-kube-api-access-s8shh\") pod \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " Oct 31 03:44:29.096285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.091841 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-metrics-certs\") pod \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " Oct 31 03:44:29.096285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.091893 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b04003a-7ae4-42c5-bb33-246aea5a0a48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b04003a-7ae4-42c5-bb33-246aea5a0a48-openshift-service-ca\") pod \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\" (UID: \"5b04003a-7ae4-42c5-bb33-246aea5a0a48\") " Oct 31 03:44:29.100074 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.100038 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b04003a-7ae4-42c5-bb33-246aea5a0a48-openshift-service-ca" pod "5b04003a-7ae4-42c5-bb33-246aea5a0a48" (UID: "5b04003a-7ae4-42c5-bb33-246aea5a0a48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:29.102535 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.102506 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls" pod "5b04003a-7ae4-42c5-bb33-246aea5a0a48" (UID: "5b04003a-7ae4-42c5-bb33-246aea5a0a48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:44:29.104757 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.104733 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04003a-7ae4-42c5-bb33-246aea5a0a48-kube-api-access-s8shh" pod "5b04003a-7ae4-42c5-bb33-246aea5a0a48" (UID: "5b04003a-7ae4-42c5-bb33-246aea5a0a48"). InnerVolumeSpecName "kube-api-access-s8shh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:29.108317 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.108288 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-metrics-certs" pod "5b04003a-7ae4-42c5-bb33-246aea5a0a48" (UID: "5b04003a-7ae4-42c5-bb33-246aea5a0a48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:44:29.199031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.198944 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.199031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.198973 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8shh\" (UniqueName: \"kubernetes.io/projected/5b04003a-7ae4-42c5-bb33-246aea5a0a48-kube-api-access-s8shh\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.199031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.198984 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b04003a-7ae4-42c5-bb33-246aea5a0a48-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.199031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.198993 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b04003a-7ae4-42c5-bb33-246aea5a0a48-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.353257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.352411 4934 generic.go:356] "Generic (PLEG): container finished" podID="5b04003a-7ae4-42c5-bb33-246aea5a0a48" containerID="630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8" exitCode=0 Oct 31 03:44:29.354255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.353536 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" Oct 31 03:44:29.381251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.379315 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" event={"ID":"5b04003a-7ae4-42c5-bb33-246aea5a0a48","Type":"ContainerDied","Data":"630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8"} Oct 31 03:44:29.381251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.379398 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" event={"ID":"5b04003a-7ae4-42c5-bb33-246aea5a0a48","Type":"ContainerDied","Data":"233f8028d194a6af7bd547a25f192c23b79327fb125b65ea3572bb4145284fd0"} Oct 31 03:44:29.381251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.379434 4934 scope.go:122] "RemoveContainer" containerID="630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8" Oct 31 03:44:29.459721 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.459420 4934 scope.go:122] "RemoveContainer" containerID="630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8" Oct 31 03:44:29.459883 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:44:29.459835 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8\": container with ID starting with 630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8 not found: ID does not exist" containerID="630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8" Oct 31 03:44:29.459955 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.459872 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8"} err="failed to get container status \"630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8\": rpc error: code = NotFound desc = could not find container \"630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8\": container with ID starting with 630a81058d24d773aa4bcb5b6f369ff96d3a891a71441adb8ca83860567a00a8 not found: ID does not exist" Oct 31 03:44:29.565254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.563850 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx"] Oct 31 03:44:29.571732 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:29.571654 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx"] Oct 31 03:44:30.823543 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:30.823508 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b04003a-7ae4-42c5-bb33-246aea5a0a48" path="/var/lib/kubelet/pods/5b04003a-7ae4-42c5-bb33-246aea5a0a48/volumes" Oct 31 03:44:31.948982 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:31.946333 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67b1ef6f-5c654bb9b-zptzx" podUID="5b04003a-7ae4-42c5-bb33-246aea5a0a48" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.71:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:44:32.660385 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:32.660348 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp"] Oct 31 03:44:32.660713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:32.660683 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" containerID="cri-o://a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e" gracePeriod=30 Oct 31 03:44:37.507334 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:37.507298 4934 patch_prober.go:29] interesting pod/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:44:37.507688 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:37.507353 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:44:37.947806 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:37.947782 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:44:38.007960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.007937 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-credential-socket\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008088 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.007974 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-data\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008088 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008056 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-kube-api-access-tj4x5\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-kube-api-access-tj4x5\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008098 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44288212-950a-4166-958a-79a131755ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44288212-950a-4166-958a-79a131755ca5-istio-podinfo\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008120 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-envoy\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008158 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-socket\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008170 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-credential-socket" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:38.008209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008194 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44288212-950a-4166-958a-79a131755ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44288212-950a-4166-958a-79a131755ca5-istiod-ca-cert\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008239 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-istio-token\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008288 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-certs\") pod \"44288212-950a-4166-958a-79a131755ca5\" (UID: \"44288212-950a-4166-958a-79a131755ca5\") " Oct 31 03:44:38.008493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008321 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-data" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:38.008628 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008600 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-certs" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:38.008628 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.008608 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-socket" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:38.009099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.009046 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.009099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.009072 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.009099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.009086 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.009099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.009099 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.009381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.009103 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44288212-950a-4166-958a-79a131755ca5-istiod-ca-cert" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:38.010688 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.010665 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-istio-token" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:38.010822 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.010789 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44288212-950a-4166-958a-79a131755ca5-istio-podinfo" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:38.010948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.010934 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-kube-api-access-tj4x5" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "kube-api-access-tj4x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:38.011011 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.010968 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-envoy" pod "44288212-950a-4166-958a-79a131755ca5" (UID: "44288212-950a-4166-958a-79a131755ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:38.110663 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.110639 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4x5\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-kube-api-access-tj4x5\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.110663 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.110661 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44288212-950a-4166-958a-79a131755ca5-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.110819 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.110671 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44288212-950a-4166-958a-79a131755ca5-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.110819 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.110679 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44288212-950a-4166-958a-79a131755ca5-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.110819 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.110688 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44288212-950a-4166-958a-79a131755ca5-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:38.609087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.609047 4934 generic.go:356] "Generic (PLEG): container finished" podID="44288212-950a-4166-958a-79a131755ca5" containerID="a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e" exitCode=0 Oct 31 03:44:38.609597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.609135 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" event={"ID":"44288212-950a-4166-958a-79a131755ca5","Type":"ContainerDied","Data":"a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e"} Oct 31 03:44:38.609597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.609247 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" event={"ID":"44288212-950a-4166-958a-79a131755ca5","Type":"ContainerDied","Data":"b1a6a9bf99f4de8744273c662058dd5df121be7048e45c749c9d27ee9a53e0c0"} Oct 31 03:44:38.609597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.609289 4934 scope.go:122] "RemoveContainer" containerID="a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e" Oct 31 03:44:38.609597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.609484 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp" Oct 31 03:44:38.647896 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.638647 4934 scope.go:122] "RemoveContainer" containerID="88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f" Oct 31 03:44:38.780616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.776449 4934 scope.go:122] "RemoveContainer" containerID="a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e" Oct 31 03:44:38.780616 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:44:38.777697 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e\": container with ID starting with a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e not found: ID does not exist" containerID="a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e" Oct 31 03:44:38.780616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.777800 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e"} err="failed to get container status \"a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e\": rpc error: code = NotFound desc = could not find container \"a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e\": container with ID starting with a70d43fb49f062412cbb494f637f8fa39acdd60e585128f332f37c13d7ac510e not found: ID does not exist" Oct 31 03:44:38.780616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.780307 4934 scope.go:122] "RemoveContainer" containerID="88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f" Oct 31 03:44:38.780919 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:44:38.780891 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f\": container with ID starting with 88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f not found: ID does not exist" containerID="88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f" Oct 31 03:44:38.781075 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.781057 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f"} err="failed to get container status \"88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f\": rpc error: code = NotFound desc = could not find container \"88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f\": container with ID starting with 88976b006acdc971eb2bf1c99736cbd5190f288bb9e40a069e3f3dcd25029d0f not found: ID does not exist" Oct 31 03:44:38.791194 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.791154 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp"] Oct 31 03:44:38.795555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.795532 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-67b1ef6f-openshift-default-7fcf446b7f-dfshp"] Oct 31 03:44:38.822436 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:44:38.822415 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44288212-950a-4166-958a-79a131755ca5" path="/var/lib/kubelet/pods/44288212-950a-4166-958a-79a131755ca5/volumes" Oct 31 03:44:49.168386 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:44:49.168344 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:45:00.171272 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.171215 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f"] Oct 31 03:45:00.172144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.172048 4934 cpu_manager.go:414] "containerMap: removing container" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" Oct 31 03:45:00.172144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.172068 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" Oct 31 03:45:00.172144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.172132 4934 cpu_manager.go:414] "containerMap: removing container" podUID="5b04003a-7ae4-42c5-bb33-246aea5a0a48" containerName="maas-api" Oct 31 03:45:00.172144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.172139 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b04003a-7ae4-42c5-bb33-246aea5a0a48" containerName="maas-api" Oct 31 03:45:00.172393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.172282 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" Oct 31 03:45:00.172393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.172295 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b04003a-7ae4-42c5-bb33-246aea5a0a48" containerName="maas-api" Oct 31 03:45:00.172393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.172306 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" Oct 31 03:45:00.175136 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.175117 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.233311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.233281 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.233416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.233338 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.233515 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.233491 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.233568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.233557 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpv6z\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-kube-api-access-bpv6z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.233636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.233618 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.233670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.233654 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.233750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.233738 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaeba949-b325-4687-ad25-c03f4a0d14a8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.274554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.274508 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f"] Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.336518 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaeba949-b325-4687-ad25-c03f4a0d14a8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.336622 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.336674 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.336811 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.336888 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv6z\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-kube-api-access-bpv6z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.336928 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.336966 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.344851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.337798 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.584347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.567831 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.584347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.583673 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.584347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.584208 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.588302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.584637 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.588302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.584958 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaeba949-b325-4687-ad25-c03f4a0d14a8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.608488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.608453 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpv6z\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-kube-api-access-bpv6z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rrq6f\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:00.788738 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:00.788709 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:01.228997 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:01.228918 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f"] Oct 31 03:45:01.265013 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:45:01.264986 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaeba949_b325_4687_ad25_c03f4a0d14a8.slice/crio-117a626e498da17ce7e9f5894a7a95f334b903b7fc6b2ecf245346b4b17a2f23 WatchSource:0}: Error finding container 117a626e498da17ce7e9f5894a7a95f334b903b7fc6b2ecf245346b4b17a2f23: Status 404 returned error can't find the container with id 117a626e498da17ce7e9f5894a7a95f334b903b7fc6b2ecf245346b4b17a2f23 Oct 31 03:45:01.267521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:01.267486 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:01.267619 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:01.267562 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:01.820623 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:01.820585 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" event={"ID":"aaeba949-b325-4687-ad25-c03f4a0d14a8","Type":"ContainerStarted","Data":"9f541eda6b237c8b272cb130d7c874f698ce41b8efd71fabfddea4caf1da85b3"} Oct 31 03:45:01.820793 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:01.820641 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" event={"ID":"aaeba949-b325-4687-ad25-c03f4a0d14a8","Type":"ContainerStarted","Data":"117a626e498da17ce7e9f5894a7a95f334b903b7fc6b2ecf245346b4b17a2f23"} Oct 31 03:45:01.820793 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:01.820668 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:02.041056 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:02.041000 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" podStartSLOduration=2.040986605 podStartE2EDuration="2.040986605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.990935281 +0000 UTC m=+1535.309677909" watchObservedRunningTime="2026-10-31 03:45:02.040986605 +0000 UTC m=+1535.359729231" Oct 31 03:45:02.833884 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:02.833850 4934 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rrq6f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:45:02.834408 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:02.833901 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" podUID="aaeba949-b325-4687-ad25-c03f4a0d14a8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:45:05.833312 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:05.833262 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:45:15.151679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.151642 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb"] Oct 31 03:45:15.153160 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.153134 4934 cpu_manager.go:414] "containerMap: removing container" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" Oct 31 03:45:15.153328 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.153314 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44288212-950a-4166-958a-79a131755ca5" containerName="istio-proxy" Oct 31 03:45:15.162891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.162866 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.200331 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.200300 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s"] Oct 31 03:45:15.204178 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.204158 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.215434 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.215409 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb"] Oct 31 03:45:15.302264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.280627 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s"] Oct 31 03:45:15.325594 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325556 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.325789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325616 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.325789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325710 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.325789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325753 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.325789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325776 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.325940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325893 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.325974 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325950 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.326007 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.325977 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.326042 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326009 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.326073 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326063 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.326105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326089 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggxx\" (UniqueName: \"kubernetes.io/projected/5453941c-2d09-4ff9-aaaf-cc0e162876e0-kube-api-access-mggxx\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.326148 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326127 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.326181 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326157 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.326257 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326211 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.326300 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326270 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.326332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326299 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.326332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326324 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.326458 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.326443 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6njz\" (UniqueName: \"kubernetes.io/projected/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-kube-api-access-q6njz\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.427690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427603 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.427690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427637 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.427690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427658 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427698 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427727 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggxx\" (UniqueName: \"kubernetes.io/projected/5453941c-2d09-4ff9-aaaf-cc0e162876e0-kube-api-access-mggxx\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427756 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427783 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427836 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427866 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427896 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.427960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427921 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.427967 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428041 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6njz\" (UniqueName: \"kubernetes.io/projected/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-kube-api-access-q6njz\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428137 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428174 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428201 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428266 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428289 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428325 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428367 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.428416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428387 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428482 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.428900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428606 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428626 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.428900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428782 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.428900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.428841 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.429173 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.429066 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.429173 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.429120 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.431946 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.431453 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.431946 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.431898 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.432661 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.432630 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.432958 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.432695 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.490594 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.490563 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5453941c-2d09-4ff9-aaaf-cc0e162876e0-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.490774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.490752 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggxx\" (UniqueName: \"kubernetes.io/projected/5453941c-2d09-4ff9-aaaf-cc0e162876e0-kube-api-access-mggxx\") pod \"maas-default-gateway-openshift-default-fd4b99879-s5kcb\" (UID: \"5453941c-2d09-4ff9-aaaf-cc0e162876e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.490955 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.490937 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.492453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.492430 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6njz\" (UniqueName: \"kubernetes.io/projected/a2e58daa-d4a4-45a5-b9ae-d7906f70c916-kube-api-access-q6njz\") pod \"maas-default-gateway-openshift-default-fd4b99879-kll7s\" (UID: \"a2e58daa-d4a4-45a5-b9ae-d7906f70c916\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.519908 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.519885 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:15.783747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.783657 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:15.964103 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.964072 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s"] Oct 31 03:45:15.968017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.967983 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:15.968128 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.968052 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:15.968128 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:15.968078 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:16.169940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.169913 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb"] Oct 31 03:45:16.172833 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:45:16.172805 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5453941c_2d09_4ff9_aaaf_cc0e162876e0.slice/crio-deff7754f827d75e13452ef899b1674da6933b464c7008f435e38b7cf29cf340 WatchSource:0}: Error finding container deff7754f827d75e13452ef899b1674da6933b464c7008f435e38b7cf29cf340: Status 404 returned error can't find the container with id deff7754f827d75e13452ef899b1674da6933b464c7008f435e38b7cf29cf340 Oct 31 03:45:16.177161 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.177123 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:16.177424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.177401 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:16.177571 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.177555 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:45:16.979793 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.979746 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" event={"ID":"a2e58daa-d4a4-45a5-b9ae-d7906f70c916","Type":"ContainerStarted","Data":"a15632786d17b39274c324b689b181a0a03fe7dfd35cda7945882ef330102dd9"} Oct 31 03:45:16.979992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.979825 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" event={"ID":"a2e58daa-d4a4-45a5-b9ae-d7906f70c916","Type":"ContainerStarted","Data":"ce850372d9ae7c0933bef00e5c28fd63f6c5ab733771f959efa0f5b312413e86"} Oct 31 03:45:16.981752 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.981724 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" event={"ID":"5453941c-2d09-4ff9-aaaf-cc0e162876e0","Type":"ContainerStarted","Data":"02670fb60cb732e6bf6b1432220dbb41bed331f979d60272074ebf30d44b31a5"} Oct 31 03:45:16.981879 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:16.981768 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" event={"ID":"5453941c-2d09-4ff9-aaaf-cc0e162876e0","Type":"ContainerStarted","Data":"deff7754f827d75e13452ef899b1674da6933b464c7008f435e38b7cf29cf340"} Oct 31 03:45:17.154283 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.154072 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podStartSLOduration=2.154053796 podStartE2EDuration="2.154053796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:17.152843526 +0000 UTC m=+1550.471586155" watchObservedRunningTime="2026-10-31 03:45:17.154053796 +0000 UTC m=+1550.472796424" Oct 31 03:45:17.278305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.277289 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podStartSLOduration=2.277268474 podStartE2EDuration="2.277268474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:17.276684647 +0000 UTC m=+1550.595427276" watchObservedRunningTime="2026-10-31 03:45:17.277268474 +0000 UTC m=+1550.596011101" Oct 31 03:45:17.522290 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.521545 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:17.524750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.524382 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:17.524750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.524448 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:17.802252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.801567 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:17.823267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.820370 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:17.823267 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:17.820450 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:18.655255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:18.641314 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:18.655255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:18.641368 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:18.789253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:18.788759 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:18.789253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:18.788819 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:19.331252 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:45:19.325545 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:45:19.524252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:19.521649 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:19.524252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:19.521720 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:19.874434 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:19.873346 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:19.874434 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:19.873417 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:20.535251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:20.534621 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:20.535251 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:20.534681 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:20.819621 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:20.819571 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:20.819933 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:20.819886 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:21.522714 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:21.522678 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:21.523326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:21.523294 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:21.808303 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:21.787046 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:21.808303 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:21.787109 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:22.521181 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:22.521139 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:22.521404 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:22.521208 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:22.785308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:22.784548 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:22.785308 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:22.784607 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:23.521252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:23.520603 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:23.521252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:23.520666 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:23.784307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:23.784134 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:23.784307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:23.784193 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:24.523846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:24.523813 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:45:24.524490 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:24.524443 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 03:45:24.784616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:24.784526 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:45:24.784616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:24.784594 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:45:26.393674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:26.393643 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:26.394314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:26.394294 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:26.395544 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:26.395523 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" Oct 31 03:45:26.697474 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:26.697386 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:26.697762 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:26.697735 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:26.699510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:26.699489 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" Oct 31 03:45:49.166736 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:45:49.166647 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:45:59.354422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:59.354375 4934 patch_prober.go:29] interesting pod/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:45:59.354920 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:45:59.354461 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:01.633078 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:01.633047 4934 generic.go:356] "Generic (PLEG): container finished" podID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerID="3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0" exitCode=0 Oct 31 03:46:01.633452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:01.633113 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" event={"ID":"d149a2e7-c176-4787-8f1a-ba47853de5d2","Type":"ContainerDied","Data":"3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0"} Oct 31 03:46:01.633559 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:01.633543 4934 scope.go:122] "RemoveContainer" containerID="3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0" Oct 31 03:46:01.635168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:01.635143 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:46:01.635293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:01.635268 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:46:01.635343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:01.635303 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:46:02.641924 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:02.641891 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" event={"ID":"d149a2e7-c176-4787-8f1a-ba47853de5d2","Type":"ContainerStarted","Data":"62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613"} Oct 31 03:46:02.975708 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:02.975623 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:46:02.976055 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:02.976028 4934 patch_prober.go:29] interesting pod/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:46:02.976201 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:02.976178 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:46:03.962587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:03.962556 4934 patch_prober.go:29] interesting pod/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:46:03.963152 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:03.963121 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:46:04.956638 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:04.956602 4934 patch_prober.go:29] interesting pod/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:46:04.956914 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:04.956884 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:46:05.965249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:05.957514 4934 patch_prober.go:29] interesting pod/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:46:05.965249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:05.957568 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:46:06.967261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:06.965595 4934 patch_prober.go:29] interesting pod/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:46:06.967261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:06.965670 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:46:07.959654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:07.959625 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:46:07.960016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:07.959996 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:46:07.960657 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:46:07.960639 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:46:19.164699 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:46:19.164657 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:46:49.248467 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:46:49.248428 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:47:17.797804 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:17.797763 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6"] Oct 31 03:47:17.798917 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:17.798879 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" containerID="cri-o://62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613" gracePeriod=30 Oct 31 03:47:19.188378 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:47:19.188299 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:47:22.962171 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:22.962114 4934 patch_prober.go:29] interesting pod/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:47:22.962642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:22.962197 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:47:23.083489 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.083422 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:47:23.170647 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.170618 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-podinfo\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.170829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.170682 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-kube-api-access-pjx6q\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-kube-api-access-pjx6q\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.170829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.170724 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d149a2e7-c176-4787-8f1a-ba47853de5d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d149a2e7-c176-4787-8f1a-ba47853de5d2-istiod-ca-cert\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.170829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.170760 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-certs\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.170829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.170790 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-credential-socket\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.171054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.170833 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-envoy\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.171054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.170985 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-data\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.171162 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171112 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d149a2e7-c176-4787-8f1a-ba47853de5d2-istiod-ca-cert" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:23.171162 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171145 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-token\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-token\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.171343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171301 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-certs" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.171343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171305 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-socket\") pod \"d149a2e7-c176-4787-8f1a-ba47853de5d2\" (UID: \"d149a2e7-c176-4787-8f1a-ba47853de5d2\") " Oct 31 03:47:23.171443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171332 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-credential-socket" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.171930 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171514 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-socket" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.171930 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171584 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-data" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.172095 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171956 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.172095 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171972 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d149a2e7-c176-4787-8f1a-ba47853de5d2-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.172095 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171984 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.172095 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.171997 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.172095 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.172006 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.173443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.173417 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-kube-api-access-pjx6q" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "kube-api-access-pjx6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:23.173600 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.173575 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-podinfo" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:47:23.174025 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.173998 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-envoy" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.174468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.174397 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-token" pod "d149a2e7-c176-4787-8f1a-ba47853de5d2" (UID: "d149a2e7-c176-4787-8f1a-ba47853de5d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:23.274167 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.274097 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.274167 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.274158 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.274398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.274175 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjx6q\" (UniqueName: \"kubernetes.io/projected/d149a2e7-c176-4787-8f1a-ba47853de5d2-kube-api-access-pjx6q\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.274398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.274190 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d149a2e7-c176-4787-8f1a-ba47853de5d2-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.852500 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.852471 4934 generic.go:356] "Generic (PLEG): container finished" podID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerID="62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613" exitCode=0 Oct 31 03:47:23.852671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.852519 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" Oct 31 03:47:23.852671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.852540 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" event={"ID":"d149a2e7-c176-4787-8f1a-ba47853de5d2","Type":"ContainerDied","Data":"62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613"} Oct 31 03:47:23.852671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.852578 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6" event={"ID":"d149a2e7-c176-4787-8f1a-ba47853de5d2","Type":"ContainerDied","Data":"88ea422f8c0ee1c0a1914ac1af2b0167b3c0cc58968ceaab887273839f84f893"} Oct 31 03:47:23.852671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.852599 4934 scope.go:122] "RemoveContainer" containerID="62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613" Oct 31 03:47:23.863634 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.863612 4934 scope.go:122] "RemoveContainer" containerID="3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0" Oct 31 03:47:23.873366 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.873345 4934 scope.go:122] "RemoveContainer" containerID="62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613" Oct 31 03:47:23.873684 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:47:23.873660 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613\": container with ID starting with 62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613 not found: ID does not exist" containerID="62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613" Oct 31 03:47:23.873733 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.873697 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613"} err="failed to get container status \"62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613\": rpc error: code = NotFound desc = could not find container \"62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613\": container with ID starting with 62339d374b5610c6cdb1250eb4d2833babd7d1387bb512a1ed283a7507543613 not found: ID does not exist" Oct 31 03:47:23.873733 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.873722 4934 scope.go:122] "RemoveContainer" containerID="3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0" Oct 31 03:47:23.873985 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:47:23.873965 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0\": container with ID starting with 3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0 not found: ID does not exist" containerID="3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0" Oct 31 03:47:23.874076 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:23.873989 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0"} err="failed to get container status \"3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0\": rpc error: code = NotFound desc = could not find container \"3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0\": container with ID starting with 3fe9e94ac51c8ff98324f7fffa5b254b390576c4b9a55250ed894214cb3647e0 not found: ID does not exist" Oct 31 03:47:24.020916 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:24.020884 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6"] Oct 31 03:47:24.072007 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:24.071976 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bb362142-gw-openshift-default-766f7659d9-x57q6"] Oct 31 03:47:24.822699 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:47:24.822667 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" path="/var/lib/kubelet/pods/d149a2e7-c176-4787-8f1a-ba47853de5d2/volumes" Oct 31 03:47:49.164189 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:47:49.164156 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:48:14.797380 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:14.797349 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs"] Oct 31 03:48:14.797908 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:14.797580 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" containerID="cri-o://0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326" gracePeriod=30 Oct 31 03:48:19.167315 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:48:19.167263 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:48:19.274062 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:19.274029 4934 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:48:19.274274 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:19.274108 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:48:20.071396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.071334 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:48:20.097417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097392 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-kube-api-access-vvztj\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-kube-api-access-vvztj\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097503 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-data\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097622 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097566 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istiod-ca-cert\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097622 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097594 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-credential-socket\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097634 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-certs\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097728 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-socket\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097757 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-podinfo\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097882 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097813 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-data" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.097882 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097863 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-token\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-token\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.097982 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097896 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istiod-ca-cert" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:20.097982 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.097922 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-envoy\") pod \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\" (UID: \"0abf3a24-37a8-422d-8b1b-d14a65fb903a\") " Oct 31 03:48:20.098216 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098189 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-certs" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.098216 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098200 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-socket" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.098850 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098423 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-credential-socket" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.098850 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098780 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.098850 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098798 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.098850 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098809 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.098850 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098817 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.098850 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.098826 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.100396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.100372 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-podinfo" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:20.100477 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.100450 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-kube-api-access-vvztj" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "kube-api-access-vvztj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.100607 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.100578 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-token" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.100817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.100794 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-envoy" pod "0abf3a24-37a8-422d-8b1b-d14a65fb903a" (UID: "0abf3a24-37a8-422d-8b1b-d14a65fb903a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.200037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.200009 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.200037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.200038 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.200441 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.200051 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0abf3a24-37a8-422d-8b1b-d14a65fb903a-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.200441 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.200063 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvztj\" (UniqueName: \"kubernetes.io/projected/0abf3a24-37a8-422d-8b1b-d14a65fb903a-kube-api-access-vvztj\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.736934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.736900 4934 generic.go:356] "Generic (PLEG): container finished" podID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerID="0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326" exitCode=0 Oct 31 03:48:20.737117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.736949 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" Oct 31 03:48:20.737117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.736988 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" event={"ID":"0abf3a24-37a8-422d-8b1b-d14a65fb903a","Type":"ContainerDied","Data":"0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326"} Oct 31 03:48:20.737117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.737053 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs" event={"ID":"0abf3a24-37a8-422d-8b1b-d14a65fb903a","Type":"ContainerDied","Data":"90ccd5c0151679870240cfd5a058b6eebb9b24eb0e28fca78fcfd4af98d9ea4c"} Oct 31 03:48:20.737117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.737080 4934 scope.go:122] "RemoveContainer" containerID="0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326" Oct 31 03:48:20.747718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.747702 4934 scope.go:122] "RemoveContainer" containerID="0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326" Oct 31 03:48:20.747985 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:48:20.747968 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326\": container with ID starting with 0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326 not found: ID does not exist" containerID="0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326" Oct 31 03:48:20.748032 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.747995 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326"} err="failed to get container status \"0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326\": rpc error: code = NotFound desc = could not find container \"0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326\": container with ID starting with 0ca763ca877ebc9e68c68e95c21fd9997e93a78a59731990cca265b21995b326 not found: ID does not exist" Oct 31 03:48:20.961177 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:20.961134 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs"] Oct 31 03:48:21.011020 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:21.010939 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-hzkjs"] Oct 31 03:48:22.827749 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:48:22.827708 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" path="/var/lib/kubelet/pods/0abf3a24-37a8-422d-8b1b-d14a65fb903a/volumes" Oct 31 03:48:49.169921 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:48:49.169880 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:49:19.385469 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:49:19.385408 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:49:24.700551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:24.700513 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp"] Oct 31 03:49:24.751256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:24.751209 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h"] Oct 31 03:49:27.594825 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:27.594797 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:49:27.663772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:27.663743 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:49:39.721875 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:39.721807 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" containerID="cri-o://4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2" gracePeriod=45 Oct 31 03:49:39.756034 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:39.755708 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": dial tcp 10.132.0.55:8000: connect: connection refused" Oct 31 03:49:39.794498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:39.794454 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" containerID="cri-o://750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8" gracePeriod=45 Oct 31 03:49:39.992513 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:39.992418 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": EOF" Oct 31 03:49:40.161047 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.161023 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:49:40.184083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.184059 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tls-certs\" (UniqueName: \"kubernetes.io/secret/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tls-certs\") pod \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " Oct 31 03:49:40.184181 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.184117 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-dshm\") pod \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " Oct 31 03:49:40.184181 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.184148 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-model-cache\") pod \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " Oct 31 03:49:40.184328 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.184177 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-kube-api-access-nnr4h\" (UniqueName: \"kubernetes.io/projected/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-kube-api-access-nnr4h\") pod \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " Oct 31 03:49:40.184328 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.184287 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-home\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-home\") pod \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " Oct 31 03:49:40.184483 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.184460 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-model-cache" pod "2012f9a6-651d-4c14-87ca-e0ec57a5f16d" (UID: "2012f9a6-651d-4c14-87ca-e0ec57a5f16d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.184624 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.184595 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-home" pod "2012f9a6-651d-4c14-87ca-e0ec57a5f16d" (UID: "2012f9a6-651d-4c14-87ca-e0ec57a5f16d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.185141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.185102 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.185141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.185125 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.186611 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.186587 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-dshm" pod "2012f9a6-651d-4c14-87ca-e0ec57a5f16d" (UID: "2012f9a6-651d-4c14-87ca-e0ec57a5f16d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.186769 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.186749 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tls-certs" pod "2012f9a6-651d-4c14-87ca-e0ec57a5f16d" (UID: "2012f9a6-651d-4c14-87ca-e0ec57a5f16d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:49:40.186839 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.186799 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-kube-api-access-nnr4h" pod "2012f9a6-651d-4c14-87ca-e0ec57a5f16d" (UID: "2012f9a6-651d-4c14-87ca-e0ec57a5f16d"). InnerVolumeSpecName "kube-api-access-nnr4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:40.286262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.286127 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tmp-dir\") pod \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\" (UID: \"2012f9a6-651d-4c14-87ca-e0ec57a5f16d\") " Oct 31 03:49:40.286446 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.286422 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tmp-dir" pod "2012f9a6-651d-4c14-87ca-e0ec57a5f16d" (UID: "2012f9a6-651d-4c14-87ca-e0ec57a5f16d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.286774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.286757 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.286856 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.286777 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.286856 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.286792 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnr4h\" (UniqueName: \"kubernetes.io/projected/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-kube-api-access-nnr4h\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.286856 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.286805 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2012f9a6-651d-4c14-87ca-e0ec57a5f16d-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.662041 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.662015 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:49:40.692814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.692787 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-tmp-dir\") pod \"7161bd91-908a-4cee-8195-7fa4ca72d836\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " Oct 31 03:49:40.692943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.692817 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7161bd91-908a-4cee-8195-7fa4ca72d836-tls-certs\" (UniqueName: \"kubernetes.io/secret/7161bd91-908a-4cee-8195-7fa4ca72d836-tls-certs\") pod \"7161bd91-908a-4cee-8195-7fa4ca72d836\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " Oct 31 03:49:40.692943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.692897 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-home\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-home\") pod \"7161bd91-908a-4cee-8195-7fa4ca72d836\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " Oct 31 03:49:40.692943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.692930 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-dshm\") pod \"7161bd91-908a-4cee-8195-7fa4ca72d836\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " Oct 31 03:49:40.693099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693005 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-model-cache\") pod \"7161bd91-908a-4cee-8195-7fa4ca72d836\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " Oct 31 03:49:40.693099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693041 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-tmp-dir" pod "7161bd91-908a-4cee-8195-7fa4ca72d836" (UID: "7161bd91-908a-4cee-8195-7fa4ca72d836"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.693099 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693079 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7161bd91-908a-4cee-8195-7fa4ca72d836-kube-api-access-lqhff\" (UniqueName: \"kubernetes.io/projected/7161bd91-908a-4cee-8195-7fa4ca72d836-kube-api-access-lqhff\") pod \"7161bd91-908a-4cee-8195-7fa4ca72d836\" (UID: \"7161bd91-908a-4cee-8195-7fa4ca72d836\") " Oct 31 03:49:40.693292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693114 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-home" pod "7161bd91-908a-4cee-8195-7fa4ca72d836" (UID: "7161bd91-908a-4cee-8195-7fa4ca72d836"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.693358 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693295 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-model-cache" pod "7161bd91-908a-4cee-8195-7fa4ca72d836" (UID: "7161bd91-908a-4cee-8195-7fa4ca72d836"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.693966 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693941 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.694068 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693972 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.694068 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.693987 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.695298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.695259 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7161bd91-908a-4cee-8195-7fa4ca72d836-tls-certs" pod "7161bd91-908a-4cee-8195-7fa4ca72d836" (UID: "7161bd91-908a-4cee-8195-7fa4ca72d836"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:49:40.695298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.695268 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-dshm" pod "7161bd91-908a-4cee-8195-7fa4ca72d836" (UID: "7161bd91-908a-4cee-8195-7fa4ca72d836"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.695298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.695292 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7161bd91-908a-4cee-8195-7fa4ca72d836-kube-api-access-lqhff" pod "7161bd91-908a-4cee-8195-7fa4ca72d836" (UID: "7161bd91-908a-4cee-8195-7fa4ca72d836"). InnerVolumeSpecName "kube-api-access-lqhff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:40.795848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.795817 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqhff\" (UniqueName: \"kubernetes.io/projected/7161bd91-908a-4cee-8195-7fa4ca72d836-kube-api-access-lqhff\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.796254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.795852 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7161bd91-908a-4cee-8195-7fa4ca72d836-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.796254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.795869 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7161bd91-908a-4cee-8195-7fa4ca72d836-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.811839 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.811813 4934 generic.go:356] "Generic (PLEG): container finished" podID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerID="750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8" exitCode=0 Oct 31 03:49:40.811971 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.811921 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" event={"ID":"7161bd91-908a-4cee-8195-7fa4ca72d836","Type":"ContainerDied","Data":"750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8"} Oct 31 03:49:40.811971 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.811955 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" event={"ID":"7161bd91-908a-4cee-8195-7fa4ca72d836","Type":"ContainerDied","Data":"e115dfee0635f864c450322405c73cc4ea46258d7e47f66a90d94256039c3b42"} Oct 31 03:49:40.812091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.811976 4934 scope.go:122] "RemoveContainer" containerID="750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8" Oct 31 03:49:40.812091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.811928 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h" Oct 31 03:49:40.813769 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.813709 4934 generic.go:356] "Generic (PLEG): container finished" podID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerID="4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2" exitCode=0 Oct 31 03:49:40.813920 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.813860 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" event={"ID":"2012f9a6-651d-4c14-87ca-e0ec57a5f16d","Type":"ContainerDied","Data":"4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2"} Oct 31 03:49:40.813920 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.813893 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" event={"ID":"2012f9a6-651d-4c14-87ca-e0ec57a5f16d","Type":"ContainerDied","Data":"6d9a8b85b51399da46dcbce3925f5f1b614efebf5aec7b16875bd59724a78aca"} Oct 31 03:49:40.814027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.813952 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp" Oct 31 03:49:40.826174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.826091 4934 scope.go:122] "RemoveContainer" containerID="750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8" Oct 31 03:49:40.826424 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:49:40.826401 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8\": container with ID starting with 750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8 not found: ID does not exist" containerID="750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8" Oct 31 03:49:40.826510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.826439 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8"} err="failed to get container status \"750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8\": rpc error: code = NotFound desc = could not find container \"750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8\": container with ID starting with 750be4769788d0000b6a9821d803736091c4431331f45ab2e2c92f5b0e47f2c8 not found: ID does not exist" Oct 31 03:49:40.826510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.826464 4934 scope.go:122] "RemoveContainer" containerID="4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2" Oct 31 03:49:40.839746 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.839725 4934 scope.go:122] "RemoveContainer" containerID="4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2" Oct 31 03:49:40.840063 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:49:40.840039 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2\": container with ID starting with 4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2 not found: ID does not exist" containerID="4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2" Oct 31 03:49:40.840164 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.840074 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2"} err="failed to get container status \"4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2\": rpc error: code = NotFound desc = could not find container \"4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2\": container with ID starting with 4783756080f73608679a68c385ce7039cf37c116109e2de2b4462f3b3b68c9b2 not found: ID does not exist" Oct 31 03:49:40.988821 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.988788 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h"] Oct 31 03:49:40.998263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:40.998217 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-096812/premium-w4-096812-kserve-7c54dd64c6-xm75h"] Oct 31 03:49:41.128370 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:41.128296 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp"] Oct 31 03:49:41.130912 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:41.130892 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-2-w4-096812-kserve-f4f678fc6-pqfpp"] Oct 31 03:49:42.827458 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:42.827428 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" path="/var/lib/kubelet/pods/2012f9a6-651d-4c14-87ca-e0ec57a5f16d/volumes" Oct 31 03:49:42.827884 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:42.827843 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" path="/var/lib/kubelet/pods/7161bd91-908a-4cee-8195-7fa4ca72d836/volumes" Oct 31 03:49:45.695277 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:45.695212 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw"] Oct 31 03:49:45.695758 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:45.695478 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" containerID="cri-o://f74c3d0e4ea9e51dc1235c94d4325d14589279148590235fcb1fb6d1a5d97a78" gracePeriod=30 Oct 31 03:49:49.168161 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:49:49.168123 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:49:50.942072 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:50.941681 4934 generic.go:356] "Generic (PLEG): container finished" podID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerID="f74c3d0e4ea9e51dc1235c94d4325d14589279148590235fcb1fb6d1a5d97a78" exitCode=0 Oct 31 03:49:50.942072 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:50.941737 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" event={"ID":"0e4f5807-2d32-4c8d-91af-31e866dbb6ed","Type":"ContainerDied","Data":"f74c3d0e4ea9e51dc1235c94d4325d14589279148590235fcb1fb6d1a5d97a78"} Oct 31 03:49:50.992104 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:50.992081 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:49:51.075497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075467 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-credential-socket\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.075658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075543 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istiod-ca-cert\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.075658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075591 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-data\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.075830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075715 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-certs\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.075830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075745 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-socket\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.075830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075775 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-credential-socket" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:51.075830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075809 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-token\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-token\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.076035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075856 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-podinfo\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.076035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075892 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-envoy\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.076035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075924 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-kube-api-access-n2z4x\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-kube-api-access-n2z4x\") pod \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\" (UID: \"0e4f5807-2d32-4c8d-91af-31e866dbb6ed\") " Oct 31 03:49:51.076035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.075957 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-data" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:51.076248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.076033 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-socket" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:51.076248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.076219 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-certs" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:51.076579 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.076550 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.076658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.076578 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istiod-ca-cert" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:49:51.076658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.076586 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.076658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.076600 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.076658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.076613 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.078739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.078709 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-envoy" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:51.078866 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.078848 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-token" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:51.078944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.078905 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-kube-api-access-n2z4x" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "kube-api-access-n2z4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:51.080132 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.080105 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-podinfo" pod "0e4f5807-2d32-4c8d-91af-31e866dbb6ed" (UID: "0e4f5807-2d32-4c8d-91af-31e866dbb6ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:49:51.177674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.177574 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.177674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.177607 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.177674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.177622 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.177674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.177635 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2z4x\" (UniqueName: \"kubernetes.io/projected/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-kube-api-access-n2z4x\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.177674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.177650 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4f5807-2d32-4c8d-91af-31e866dbb6ed-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:51.949019 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.948989 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" Oct 31 03:49:51.949479 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.949016 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw" event={"ID":"0e4f5807-2d32-4c8d-91af-31e866dbb6ed","Type":"ContainerDied","Data":"43f1633f54643438fdeb69396951d1bc86e4bd20c2327def5b9723999990a7f7"} Oct 31 03:49:51.949479 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:51.949078 4934 scope.go:122] "RemoveContainer" containerID="f74c3d0e4ea9e51dc1235c94d4325d14589279148590235fcb1fb6d1a5d97a78" Oct 31 03:49:52.123629 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:52.123591 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw"] Oct 31 03:49:52.173033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:52.173000 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-kc9pw"] Oct 31 03:49:52.979087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:52.979053 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" path="/var/lib/kubelet/pods/0e4f5807-2d32-4c8d-91af-31e866dbb6ed/volumes" Oct 31 03:49:59.517280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:59.517243 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb"] Oct 31 03:49:59.517794 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:59.517463 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" containerID="cri-o://bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f" gracePeriod=30 Oct 31 03:49:59.822013 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:59.821983 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6"] Oct 31 03:49:59.822203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:49:59.822174 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" containerID="cri-o://f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a" gracePeriod=30 Oct 31 03:50:00.180147 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.180064 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w"] Oct 31 03:50:00.180394 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.180350 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" containerID="cri-o://36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68" gracePeriod=30 Oct 31 03:50:00.183057 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.183028 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc"] Oct 31 03:50:00.183310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.183253 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" containerID="cri-o://31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668" gracePeriod=30 Oct 31 03:50:00.185888 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.185863 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg"] Oct 31 03:50:00.186100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.186051 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" containerID="cri-o://2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780" gracePeriod=30 Oct 31 03:50:00.202453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.202158 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:50:00.202453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.202205 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:50:00.752863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.752831 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc"] Oct 31 03:50:00.753311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.753037 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" containerID="cri-o://9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e" gracePeriod=30 Oct 31 03:50:00.755495 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.755462 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z"] Oct 31 03:50:00.755673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:00.755645 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" containerID="cri-o://a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8" gracePeriod=30 Oct 31 03:50:01.731256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:01.718555 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:50:01.731256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:01.718620 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:50:01.947502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:01.944999 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:50:01.947502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:01.945061 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:50:02.843590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:02.843553 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:50:02.843765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:02.843604 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:50:04.806562 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.806532 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:50:04.892725 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.892692 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-data\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.892880 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.892861 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-kube-api-access-qb4gd\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-kube-api-access-qb4gd\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.892941 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.892905 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-data" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:04.892941 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.892912 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-podinfo\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.893037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.892984 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-envoy\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.893037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893015 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-credential-socket\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.893113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893040 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-socket\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.893113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893073 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-token\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-token\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.893193 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893120 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8115d9-3924-42f4-96ec-29ecfd711637-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8115d9-3924-42f4-96ec-29ecfd711637-istiod-ca-cert\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.893264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893205 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-certs\") pod \"cc8115d9-3924-42f4-96ec-29ecfd711637\" (UID: \"cc8115d9-3924-42f4-96ec-29ecfd711637\") " Oct 31 03:50:04.894082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893547 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-credential-socket" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:04.894082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893647 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8115d9-3924-42f4-96ec-29ecfd711637-istiod-ca-cert" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:04.894082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.893741 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-certs" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:04.894082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.894012 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.894082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.894033 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.894082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.894047 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.894082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.894062 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8115d9-3924-42f4-96ec-29ecfd711637-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.894693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.894103 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-socket" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:04.896149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.896121 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-envoy" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:04.896271 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.896153 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-podinfo" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:04.896341 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.896257 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-token" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:04.896471 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.896453 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-kube-api-access-qb4gd" pod "cc8115d9-3924-42f4-96ec-29ecfd711637" (UID: "cc8115d9-3924-42f4-96ec-29ecfd711637"). InnerVolumeSpecName "kube-api-access-qb4gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:04.995859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.995824 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.995859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.995859 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8115d9-3924-42f4-96ec-29ecfd711637-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.996276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.995874 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.996276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.995888 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb4gd\" (UniqueName: \"kubernetes.io/projected/cc8115d9-3924-42f4-96ec-29ecfd711637-kube-api-access-qb4gd\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:04.996276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:04.995904 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8115d9-3924-42f4-96ec-29ecfd711637-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.013009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.012815 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:50:05.013009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.012888 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:50:05.117495 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.117464 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:50:05.184683 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.184651 4934 generic.go:356] "Generic (PLEG): container finished" podID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerID="f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a" exitCode=0 Oct 31 03:50:05.184859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.184709 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" Oct 31 03:50:05.184859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.184738 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" event={"ID":"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4","Type":"ContainerDied","Data":"f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a"} Oct 31 03:50:05.184859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.184795 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6" event={"ID":"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4","Type":"ContainerDied","Data":"544aed8cf112f0e404a89f6fdb6d49370ddd2d219a96950c68c8692ed7e8e279"} Oct 31 03:50:05.184859 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.184824 4934 scope.go:122] "RemoveContainer" containerID="f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a" Oct 31 03:50:05.187837 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.187814 4934 generic.go:356] "Generic (PLEG): container finished" podID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerID="bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f" exitCode=0 Oct 31 03:50:05.188050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.188033 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" Oct 31 03:50:05.198888 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.198862 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-data\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.198949 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-kube-api-access-79pmt\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-kube-api-access-79pmt\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199047 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-envoy\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199091 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-credential-socket\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199119 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istiod-ca-cert\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199146 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-podinfo\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199197 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-certs\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199222 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-token\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-token\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199373 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-socket\") pod \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\" (UID: \"3b69ec0a-4977-4db7-bab5-bc5ca4a808d4\") " Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.199985 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-certs" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.200200 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-socket" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.200323 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istiod-ca-cert" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.200676 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-data" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.201245 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.201309 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-credential-socket" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.202305 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" event={"ID":"cc8115d9-3924-42f4-96ec-29ecfd711637","Type":"ContainerDied","Data":"bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f"} Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.202368 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb" event={"ID":"cc8115d9-3924-42f4-96ec-29ecfd711637","Type":"ContainerDied","Data":"92b93be64a104e43a8d2d947a4f6e4d2cb329158e74053523897e46431bea087"} Oct 31 03:50:05.203396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.202974 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-token" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:05.206514 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.206476 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-kube-api-access-79pmt" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "kube-api-access-79pmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:05.207006 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.206985 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-envoy" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.207614 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.207583 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-podinfo" pod "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" (UID: "3b69ec0a-4977-4db7-bab5-bc5ca4a808d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:05.211164 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.210873 4934 scope.go:122] "RemoveContainer" containerID="f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a" Oct 31 03:50:05.212996 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:05.211405 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a\": container with ID starting with f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a not found: ID does not exist" containerID="f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a" Oct 31 03:50:05.212996 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.211441 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a"} err="failed to get container status \"f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a\": rpc error: code = NotFound desc = could not find container \"f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a\": container with ID starting with f49665f301ac03b465e05db630997214b6192ad5d23c16059ae4ede0a71f1f1a not found: ID does not exist" Oct 31 03:50:05.212996 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.211479 4934 scope.go:122] "RemoveContainer" containerID="bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312755 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312786 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79pmt\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-kube-api-access-79pmt\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312804 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312818 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312832 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312845 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312858 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.312903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.312870 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.335034 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.335009 4934 scope.go:122] "RemoveContainer" containerID="bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f" Oct 31 03:50:05.335522 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:05.335488 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f\": container with ID starting with bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f not found: ID does not exist" containerID="bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f" Oct 31 03:50:05.335616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.335535 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f"} err="failed to get container status \"bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f\": rpc error: code = NotFound desc = could not find container \"bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f\": container with ID starting with bca0a1b8d7433c6529e8d764407d399253bf4e8bb108f4ef1626ca948957d47f not found: ID does not exist" Oct 31 03:50:05.539626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.535752 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb"] Oct 31 03:50:05.555512 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.548190 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-k9hgb"] Oct 31 03:50:05.625572 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.625459 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:50:05.727493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.727458 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:50:05.739532 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.739502 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6"] Oct 31 03:50:05.743849 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.743769 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-mcxh6"] Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.764621 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-credential-socket\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.764731 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133fb05a-c4d1-4d03-a051-61bf044287f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133fb05a-c4d1-4d03-a051-61bf044287f3-istiod-ca-cert\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.764762 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-socket\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.764861 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-data\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.764919 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-kube-api-access-wbfgm\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-kube-api-access-wbfgm\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.764946 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-token\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-token\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.764975 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-podinfo\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.765003 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-envoy\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.765122 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-certs\") pod \"133fb05a-c4d1-4d03-a051-61bf044287f3\" (UID: \"133fb05a-c4d1-4d03-a051-61bf044287f3\") " Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.765927 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-certs" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.767061 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-socket" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.767126 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-credential-socket" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.769256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.767284 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133fb05a-c4d1-4d03-a051-61bf044287f3-istiod-ca-cert" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:05.778519 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.772193 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-data" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.778519 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.776824 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-envoy" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.778519 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.776923 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-token" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:05.781478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.781440 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-kube-api-access-wbfgm" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "kube-api-access-wbfgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:05.782313 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.782272 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-podinfo" pod "133fb05a-c4d1-4d03-a051-61bf044287f3" (UID: "133fb05a-c4d1-4d03-a051-61bf044287f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:05.872841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.872630 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-credential-socket\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.872841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.872711 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-socket\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.872841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.872753 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-kube-api-access-8xbsb\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-kube-api-access-8xbsb\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.872841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.872811 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-podinfo\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.872890 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-certs\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.872939 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-envoy\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.872974 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-token\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-token\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.873049 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce11cf56-0806-4031-87a6-f5facaf9859b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11cf56-0806-4031-87a6-f5facaf9859b-istiod-ca-cert\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.873254 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-data\") pod \"ce11cf56-0806-4031-87a6-f5facaf9859b\" (UID: \"ce11cf56-0806-4031-87a6-f5facaf9859b\") " Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.873982 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874007 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133fb05a-c4d1-4d03-a051-61bf044287f3-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874022 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874035 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874050 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbfgm\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-kube-api-access-wbfgm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874065 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874079 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874101 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.875582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.874116 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133fb05a-c4d1-4d03-a051-61bf044287f3-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.879277 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.879161 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-certs" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.886271 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.886200 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-data" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.888394 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.888355 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-socket" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.888684 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.888651 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce11cf56-0806-4031-87a6-f5facaf9859b-istiod-ca-cert" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:05.888828 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.888800 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-credential-socket" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.891510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.891476 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-envoy" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:05.891948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.891918 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-token" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:05.892855 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.892806 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-podinfo" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:05.894020 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.893985 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-kube-api-access-8xbsb" pod "ce11cf56-0806-4031-87a6-f5facaf9859b" (UID: "ce11cf56-0806-4031-87a6-f5facaf9859b"). InnerVolumeSpecName "kube-api-access-8xbsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:05.959412 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.959377 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976092 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976124 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbsb\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-kube-api-access-8xbsb\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976137 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976150 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976166 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976178 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976189 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce11cf56-0806-4031-87a6-f5facaf9859b-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976201 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:05.976659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:05.976213 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce11cf56-0806-4031-87a6-f5facaf9859b-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.082748 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-envoy\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.082799 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istiod-ca-cert\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.082854 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-socket\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.082897 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-certs\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.082961 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-podinfo\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.083043 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-token\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-token\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.083149 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-data\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.083204 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-credential-socket\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.084168 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.083289 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-kube-api-access-l697w\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-kube-api-access-l697w\") pod \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\" (UID: \"50f391f7-fcac-4181-aeb6-11a9b5e27eb0\") " Oct 31 03:50:06.094217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.090032 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istiod-ca-cert" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:06.094217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.090382 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-socket" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.094217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.093859 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-data" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.097126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.097028 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-certs" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.100447 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.100333 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-kube-api-access-l697w" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "kube-api-access-l697w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.100988 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.100954 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-envoy" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.101091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.101042 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-token" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.101457 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.101410 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-credential-socket" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.101954 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.101931 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-podinfo" pod "50f391f7-fcac-4181-aeb6-11a9b5e27eb0" (UID: "50f391f7-fcac-4181-aeb6-11a9b5e27eb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:06.112726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.112677 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:50:06.147998 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.147965 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:50:06.185254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185198 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-socket\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185337 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-kube-api-access-jrb7f\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-kube-api-access-jrb7f\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185387 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e14a0a-db97-451a-b942-171d5d643415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e14a0a-db97-451a-b942-171d5d643415-istiod-ca-cert\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185544 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185463 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-socket" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.185544 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185485 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-credential-socket\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185547 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-envoy\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185608 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2e14a0a-db97-451a-b942-171d5d643415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e14a0a-db97-451a-b942-171d5d643415-istio-podinfo\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185658 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-data\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185693 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-certs\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185727 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-istio-token\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-istio-token\") pod \"a2e14a0a-db97-451a-b942-171d5d643415\" (UID: \"a2e14a0a-db97-451a-b942-171d5d643415\") " Oct 31 03:50:06.185884 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.185807 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e14a0a-db97-451a-b942-171d5d643415-istiod-ca-cert" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186158 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-data" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186286 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-credential-socket" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186297 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186331 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186348 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186362 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186376 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186391 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186405 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186419 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e14a0a-db97-451a-b942-171d5d643415-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186433 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186448 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186460 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l697w\" (UniqueName: \"kubernetes.io/projected/50f391f7-fcac-4181-aeb6-11a9b5e27eb0-kube-api-access-l697w\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.186494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.186474 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.187321 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.187298 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-certs" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.189907 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.188611 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-kube-api-access-jrb7f" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "kube-api-access-jrb7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.190008 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.189947 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-istio-token" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.190345 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.190308 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2e14a0a-db97-451a-b942-171d5d643415-istio-podinfo" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:06.190528 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.190489 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-envoy" pod "a2e14a0a-db97-451a-b942-171d5d643415" (UID: "a2e14a0a-db97-451a-b942-171d5d643415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.197857 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.197829 4934 generic.go:356] "Generic (PLEG): container finished" podID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerID="a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8" exitCode=0 Oct 31 03:50:06.197969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.197947 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" Oct 31 03:50:06.198043 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.197946 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" event={"ID":"a45a1814-9960-4ad4-9f1f-4991f4558d02","Type":"ContainerDied","Data":"a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8"} Oct 31 03:50:06.198112 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.198076 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z" event={"ID":"a45a1814-9960-4ad4-9f1f-4991f4558d02","Type":"ContainerDied","Data":"91d2940f92f70506ed9742c04e1cf941edafe2b750643eed80af76c1b621d494"} Oct 31 03:50:06.198186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.198116 4934 scope.go:122] "RemoveContainer" containerID="a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8" Oct 31 03:50:06.200047 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.200023 4934 generic.go:356] "Generic (PLEG): container finished" podID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerID="2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780" exitCode=0 Oct 31 03:50:06.200158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.200069 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" Oct 31 03:50:06.200216 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.200161 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" event={"ID":"50f391f7-fcac-4181-aeb6-11a9b5e27eb0","Type":"ContainerDied","Data":"2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780"} Oct 31 03:50:06.200316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.200260 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg" event={"ID":"50f391f7-fcac-4181-aeb6-11a9b5e27eb0","Type":"ContainerDied","Data":"4745f068045599001d27fa5b1d45fa0a4ba06f300ced55f0e4d29358614cb3fd"} Oct 31 03:50:06.201996 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.201978 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" Oct 31 03:50:06.202344 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.202318 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" event={"ID":"133fb05a-c4d1-4d03-a051-61bf044287f3","Type":"ContainerDied","Data":"36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68"} Oct 31 03:50:06.208197 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.208172 4934 generic.go:356] "Generic (PLEG): container finished" podID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerID="36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68" exitCode=0 Oct 31 03:50:06.208742 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.208417 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w" event={"ID":"133fb05a-c4d1-4d03-a051-61bf044287f3","Type":"ContainerDied","Data":"6c95aa0a64a8da9cf51d5eaf4f5d9e2269a140d03840e7317d8c196ba4f9be3b"} Oct 31 03:50:06.210393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.210375 4934 generic.go:356] "Generic (PLEG): container finished" podID="a2e14a0a-db97-451a-b942-171d5d643415" containerID="9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e" exitCode=0 Oct 31 03:50:06.210510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.210489 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" event={"ID":"a2e14a0a-db97-451a-b942-171d5d643415","Type":"ContainerDied","Data":"9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e"} Oct 31 03:50:06.210576 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.210532 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" event={"ID":"a2e14a0a-db97-451a-b942-171d5d643415","Type":"ContainerDied","Data":"a8a52d7ca19ed067a89b08355fec62b4d0aa5ee149af0e38dcd6f203fbbb41ee"} Oct 31 03:50:06.210630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.210604 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc" Oct 31 03:50:06.212046 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.212028 4934 scope.go:122] "RemoveContainer" containerID="a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8" Oct 31 03:50:06.212440 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:06.212415 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8\": container with ID starting with a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8 not found: ID does not exist" containerID="a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8" Oct 31 03:50:06.212541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.212451 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8"} err="failed to get container status \"a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8\": rpc error: code = NotFound desc = could not find container \"a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8\": container with ID starting with a39763abe165f97b63970012e9fcd83e695f6edbf9540cbdeb2130a1d849e0b8 not found: ID does not exist" Oct 31 03:50:06.212541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.212472 4934 scope.go:122] "RemoveContainer" containerID="2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780" Oct 31 03:50:06.215750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.215641 4934 generic.go:356] "Generic (PLEG): container finished" podID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerID="31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668" exitCode=0 Oct 31 03:50:06.215750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.215667 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" event={"ID":"ce11cf56-0806-4031-87a6-f5facaf9859b","Type":"ContainerDied","Data":"31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668"} Oct 31 03:50:06.215750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.215686 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" Oct 31 03:50:06.215750 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.215712 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc" event={"ID":"ce11cf56-0806-4031-87a6-f5facaf9859b","Type":"ContainerDied","Data":"dbf80d2eb36bf06006a5be3d94ea9b1db3b2647f073c61f65683ef7ef6fb702d"} Oct 31 03:50:06.225978 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.225850 4934 scope.go:122] "RemoveContainer" containerID="2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780" Oct 31 03:50:06.226370 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:06.226331 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780\": container with ID starting with 2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780 not found: ID does not exist" containerID="2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780" Oct 31 03:50:06.226468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.226380 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780"} err="failed to get container status \"2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780\": rpc error: code = NotFound desc = could not find container \"2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780\": container with ID starting with 2321ccefbd16ad5c59e4e898f50de9f2f842a50e7afb4ba37dfe0cf04a364780 not found: ID does not exist" Oct 31 03:50:06.226468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.226402 4934 scope.go:122] "RemoveContainer" containerID="36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68" Oct 31 03:50:06.237125 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.237106 4934 scope.go:122] "RemoveContainer" containerID="36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68" Oct 31 03:50:06.237384 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:06.237363 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68\": container with ID starting with 36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68 not found: ID does not exist" containerID="36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68" Oct 31 03:50:06.237467 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.237399 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68"} err="failed to get container status \"36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68\": rpc error: code = NotFound desc = could not find container \"36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68\": container with ID starting with 36bab7753273072a963ea862e179d5551b84215e22b1f14cefc4b7884febfd68 not found: ID does not exist" Oct 31 03:50:06.237467 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.237421 4934 scope.go:122] "RemoveContainer" containerID="9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e" Oct 31 03:50:06.246205 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.246166 4934 scope.go:122] "RemoveContainer" containerID="9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e" Oct 31 03:50:06.246536 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:06.246511 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e\": container with ID starting with 9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e not found: ID does not exist" containerID="9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e" Oct 31 03:50:06.246639 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.246542 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e"} err="failed to get container status \"9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e\": rpc error: code = NotFound desc = could not find container \"9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e\": container with ID starting with 9d749f87ed4d0b201c357ce8c0fa28a2b6ef963771f9ae1206a4ff165988926e not found: ID does not exist" Oct 31 03:50:06.246639 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.246564 4934 scope.go:122] "RemoveContainer" containerID="31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668" Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.305723 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-certs\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.305778 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-credential-socket\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.305832 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-kube-api-access-4fz2w\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-kube-api-access-4fz2w\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.305898 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-data\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306028 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-socket\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306078 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-token\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-token\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306112 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-envoy\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306163 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45a1814-9960-4ad4-9f1f-4991f4558d02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45a1814-9960-4ad4-9f1f-4991f4558d02-istiod-ca-cert\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306269 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-podinfo\") pod \"a45a1814-9960-4ad4-9f1f-4991f4558d02\" (UID: \"a45a1814-9960-4ad4-9f1f-4991f4558d02\") " Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306801 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306823 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306840 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e14a0a-db97-451a-b942-171d5d643415-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306854 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e14a0a-db97-451a-b942-171d5d643415-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306871 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.311601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.306889 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrb7f\" (UniqueName: \"kubernetes.io/projected/a2e14a0a-db97-451a-b942-171d5d643415-kube-api-access-jrb7f\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.315540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.312281 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-data" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.315540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.312499 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-certs" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.315540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.313008 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-credential-socket" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.315540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.314513 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-socket" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.315540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.314855 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45a1814-9960-4ad4-9f1f-4991f4558d02-istiod-ca-cert" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:06.326206 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.325872 4934 scope.go:122] "RemoveContainer" containerID="31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668" Oct 31 03:50:06.328138 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:06.326664 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668\": container with ID starting with 31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668 not found: ID does not exist" containerID="31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668" Oct 31 03:50:06.328138 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.326701 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668"} err="failed to get container status \"31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668\": rpc error: code = NotFound desc = could not find container \"31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668\": container with ID starting with 31c8cd3c884a80850e1d66dabc0f280c47ca3177e30caf625b5abca8d97d6668 not found: ID does not exist" Oct 31 03:50:06.364828 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.364764 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-podinfo" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:06.388473 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.388390 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg"] Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.408198 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.408252 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.408270 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.408284 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.408297 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.408311 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45a1814-9960-4ad4-9f1f-4991f4558d02-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.411405 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-token" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.411490 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-envoy" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:06.418384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.417004 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-kube-api-access-4fz2w" pod "a45a1814-9960-4ad4-9f1f-4991f4558d02" (UID: "a45a1814-9960-4ad4-9f1f-4991f4558d02"). InnerVolumeSpecName "kube-api-access-4fz2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.426435 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.426404 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-m4cbg"] Oct 31 03:50:06.509740 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.509643 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.509740 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.509675 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45a1814-9960-4ad4-9f1f-4991f4558d02-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.509740 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.509690 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fz2w\" (UniqueName: \"kubernetes.io/projected/a45a1814-9960-4ad4-9f1f-4991f4558d02-kube-api-access-4fz2w\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.634711 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.634612 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc"] Oct 31 03:50:06.638584 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.638556 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-8hwhc"] Oct 31 03:50:06.738929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.738895 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc"] Oct 31 03:50:06.765391 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.765360 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-l8rmc"] Oct 31 03:50:06.941140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.941059 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" path="/var/lib/kubelet/pods/3b69ec0a-4977-4db7-bab5-bc5ca4a808d4/volumes" Oct 31 03:50:06.942654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.942628 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" path="/var/lib/kubelet/pods/50f391f7-fcac-4181-aeb6-11a9b5e27eb0/volumes" Oct 31 03:50:06.943454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.943436 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e14a0a-db97-451a-b942-171d5d643415" path="/var/lib/kubelet/pods/a2e14a0a-db97-451a-b942-171d5d643415/volumes" Oct 31 03:50:06.944593 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.944574 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" path="/var/lib/kubelet/pods/cc8115d9-3924-42f4-96ec-29ecfd711637/volumes" Oct 31 03:50:06.945269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.945252 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" path="/var/lib/kubelet/pods/ce11cf56-0806-4031-87a6-f5facaf9859b/volumes" Oct 31 03:50:06.976127 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.976096 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w"] Oct 31 03:50:06.979377 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:06.979350 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-4rs4w"] Oct 31 03:50:07.083799 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.083769 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z"] Oct 31 03:50:07.088436 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.088411 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-tqm6z"] Oct 31 03:50:07.981960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.981929 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7153-fd3fa5"] Oct 31 03:50:07.982592 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982572 4934 cpu_manager.go:414] "containerMap: removing container" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" Oct 31 03:50:07.982636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982594 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" Oct 31 03:50:07.982636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982609 4934 cpu_manager.go:414] "containerMap: removing container" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" Oct 31 03:50:07.982636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982614 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" Oct 31 03:50:07.982636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982622 4934 cpu_manager.go:414] "containerMap: removing container" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" Oct 31 03:50:07.982636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982626 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982638 4934 cpu_manager.go:414] "containerMap: removing container" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982644 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982650 4934 cpu_manager.go:414] "containerMap: removing container" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982654 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982666 4934 cpu_manager.go:414] "containerMap: removing container" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982671 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982677 4934 cpu_manager.go:414] "containerMap: removing container" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982681 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982689 4934 cpu_manager.go:414] "containerMap: removing container" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982692 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982705 4934 cpu_manager.go:414] "containerMap: removing container" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982709 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982717 4934 cpu_manager.go:414] "containerMap: removing container" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982721 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982739 4934 cpu_manager.go:414] "containerMap: removing container" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982743 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982749 4934 cpu_manager.go:414] "containerMap: removing container" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" Oct 31 03:50:07.982789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982753 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982826 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f391f7-fcac-4181-aeb6-11a9b5e27eb0" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982832 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e14a0a-db97-451a-b942-171d5d643415" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982838 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982845 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="7161bd91-908a-4cee-8195-7fa4ca72d836" containerName="main" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982850 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b69ec0a-4977-4db7-bab5-bc5ca4a808d4" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982858 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982863 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4f5807-2d32-4c8d-91af-31e866dbb6ed" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982869 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="2012f9a6-651d-4c14-87ca-e0ec57a5f16d" containerName="main" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982875 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce11cf56-0806-4031-87a6-f5facaf9859b" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982881 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="0abf3a24-37a8-422d-8b1b-d14a65fb903a" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982886 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8115d9-3924-42f4-96ec-29ecfd711637" containerName="istio-proxy" Oct 31 03:50:07.983381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.982890 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" containerName="istio-proxy" Oct 31 03:50:07.987654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:07.987637 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:08.014388 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.014363 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbgww\" (UniqueName: \"kubernetes.io/projected/d8e020b0-f83c-4c78-a7ea-aee5270092ea-kube-api-access-rbgww\") pod \"test-curl-7153-fd3fa5\" (UID: \"d8e020b0-f83c-4c78-a7ea-aee5270092ea\") " pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:08.040365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.040335 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hppm7\"" Oct 31 03:50:08.042529 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.042503 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-fd3fa5"] Oct 31 03:50:08.115417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.115389 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbgww\" (UniqueName: \"kubernetes.io/projected/d8e020b0-f83c-4c78-a7ea-aee5270092ea-kube-api-access-rbgww\") pod \"test-curl-7153-fd3fa5\" (UID: \"d8e020b0-f83c-4c78-a7ea-aee5270092ea\") " pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:08.176368 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.176338 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbgww\" (UniqueName: \"kubernetes.io/projected/d8e020b0-f83c-4c78-a7ea-aee5270092ea-kube-api-access-rbgww\") pod \"test-curl-7153-fd3fa5\" (UID: \"d8e020b0-f83c-4c78-a7ea-aee5270092ea\") " pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:08.299773 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.299687 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:08.684403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.684379 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-fd3fa5"] Oct 31 03:50:08.686367 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:50:08.686342 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8e020b0_f83c_4c78_a7ea_aee5270092ea.slice/crio-3fb249bef169dc5f986493e35022c6c5b417afd7e7a246ae2fb6dfcc4340e5fc WatchSource:0}: Error finding container 3fb249bef169dc5f986493e35022c6c5b417afd7e7a246ae2fb6dfcc4340e5fc: Status 404 returned error can't find the container with id 3fb249bef169dc5f986493e35022c6c5b417afd7e7a246ae2fb6dfcc4340e5fc Oct 31 03:50:08.688432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.688419 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:50:08.860740 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.851632 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133fb05a-c4d1-4d03-a051-61bf044287f3" path="/var/lib/kubelet/pods/133fb05a-c4d1-4d03-a051-61bf044287f3/volumes" Oct 31 03:50:08.860740 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:08.857600 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45a1814-9960-4ad4-9f1f-4991f4558d02" path="/var/lib/kubelet/pods/a45a1814-9960-4ad4-9f1f-4991f4558d02/volumes" Oct 31 03:50:09.172785 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.172754 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x"] Oct 31 03:50:09.174018 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.173995 4934 cpu_manager.go:414] "containerMap: removing container" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" Oct 31 03:50:09.174169 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.174158 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" Oct 31 03:50:09.174359 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.174347 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="d149a2e7-c176-4787-8f1a-ba47853de5d2" containerName="istio-proxy" Oct 31 03:50:09.178929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.178911 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.234410 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.234373 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x"] Oct 31 03:50:09.239219 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.239187 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-84e130-openshift-default-dockercfg-s85m9\"" Oct 31 03:50:09.247175 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.247143 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-istio-token\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.247535 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.247511 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-envoy\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.247769 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.247749 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a196b3-a77a-414a-9995-b766598757aa-istiod-ca-cert\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.247969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.247952 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a196b3-a77a-414a-9995-b766598757aa-istio-podinfo\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.248175 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.248157 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9sf\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-kube-api-access-dh9sf\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.248369 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.248351 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-socket\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.248543 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.248525 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-certs\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.248732 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.248716 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-data\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.248945 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.248921 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-credential-socket\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.249881 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.249854 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-fd3fa5" event={"ID":"d8e020b0-f83c-4c78-a7ea-aee5270092ea","Type":"ContainerStarted","Data":"3fb249bef169dc5f986493e35022c6c5b417afd7e7a246ae2fb6dfcc4340e5fc"} Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.370944 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-envoy\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371029 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a196b3-a77a-414a-9995-b766598757aa-istiod-ca-cert\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371072 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a196b3-a77a-414a-9995-b766598757aa-istio-podinfo\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371116 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9sf\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-kube-api-access-dh9sf\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371148 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-socket\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371185 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-certs\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371264 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-data\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371349 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-credential-socket\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.372269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.371395 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-istio-token\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.397253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.373950 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a196b3-a77a-414a-9995-b766598757aa-istiod-ca-cert\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.397253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.382599 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-data\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.397253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.382839 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-certs\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.397253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.383111 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-credential-socket\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.397253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.383483 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-socket\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.397660 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.397497 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-envoy\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.424900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.424817 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a196b3-a77a-414a-9995-b766598757aa-istio-podinfo\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.433679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.433648 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-istio-token\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.444004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.443970 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9sf\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-kube-api-access-dh9sf\") pod \"e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.506270 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.503924 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:09.993783 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.993749 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x"] Oct 31 03:50:09.994884 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:50:09.994854 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a196b3_a77a_414a_9995_b766598757aa.slice/crio-54478e7be7dd3ecc84ec8b19da9504b1be5567b0096335fb93c7c8978bd3deda WatchSource:0}: Error finding container 54478e7be7dd3ecc84ec8b19da9504b1be5567b0096335fb93c7c8978bd3deda: Status 404 returned error can't find the container with id 54478e7be7dd3ecc84ec8b19da9504b1be5567b0096335fb93c7c8978bd3deda Oct 31 03:50:09.998080 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.998041 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:09.998210 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.998150 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:09.998210 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:09.998194 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:10.264429 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:10.264338 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" event={"ID":"25a196b3-a77a-414a-9995-b766598757aa","Type":"ContainerStarted","Data":"0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3"} Oct 31 03:50:10.264922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:10.264439 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" event={"ID":"25a196b3-a77a-414a-9995-b766598757aa","Type":"ContainerStarted","Data":"54478e7be7dd3ecc84ec8b19da9504b1be5567b0096335fb93c7c8978bd3deda"} Oct 31 03:50:10.392769 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:10.392711 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podStartSLOduration=1.392693038 podStartE2EDuration="1.392693038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:10.385294948 +0000 UTC m=+1843.704037575" watchObservedRunningTime="2026-10-31 03:50:10.392693038 +0000 UTC m=+1843.711435667" Oct 31 03:50:10.504539 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:10.504439 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:11.511648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:11.511615 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:12.279008 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:12.278917 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-fd3fa5" event={"ID":"d8e020b0-f83c-4c78-a7ea-aee5270092ea","Type":"ContainerStarted","Data":"4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd"} Oct 31 03:50:12.279491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:12.279420 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:12.280319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:12.280299 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:50:12.394865 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:12.394808 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7153-fd3fa5" podStartSLOduration=2.181619084 podStartE2EDuration="5.394788201s" totalImagesPullingTime="3.213169117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:12.392417686 +0000 UTC m=+1845.711160313" watchObservedRunningTime="2026-10-31 03:50:12.394788201 +0000 UTC m=+1845.713530829" Oct 31 03:50:13.994436 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:13.994400 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7153-fd3fa5"] Oct 31 03:50:14.000702 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:14.000675 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7153-fd3fa5"] Oct 31 03:50:14.292145 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:14.291960 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7153-fd3fa5" podUID="d8e020b0-f83c-4c78-a7ea-aee5270092ea" containerName="test-curl-7153-fd3fa5" containerID="cri-o://4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd" gracePeriod=2 Oct 31 03:50:14.343436 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:14.343402 4934 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-fd3fa5\" is forbidden: User \"system:node:ip-10-0-138-238.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-238.us-west-1.compute.internal' and this object" podUID="d8e020b0-f83c-4c78-a7ea-aee5270092ea" pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:16.554033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:16.553976 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:16.698723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:16.697711 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e020b0-f83c-4c78-a7ea-aee5270092ea-kube-api-access-rbgww\" (UniqueName: \"kubernetes.io/projected/d8e020b0-f83c-4c78-a7ea-aee5270092ea-kube-api-access-rbgww\") pod \"d8e020b0-f83c-4c78-a7ea-aee5270092ea\" (UID: \"d8e020b0-f83c-4c78-a7ea-aee5270092ea\") " Oct 31 03:50:16.767813 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:16.767761 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e020b0-f83c-4c78-a7ea-aee5270092ea-kube-api-access-rbgww" pod "d8e020b0-f83c-4c78-a7ea-aee5270092ea" (UID: "d8e020b0-f83c-4c78-a7ea-aee5270092ea"). InnerVolumeSpecName "kube-api-access-rbgww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:16.837415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:16.802393 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbgww\" (UniqueName: \"kubernetes.io/projected/d8e020b0-f83c-4c78-a7ea-aee5270092ea-kube-api-access-rbgww\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:16.921253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:16.892944 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e020b0-f83c-4c78-a7ea-aee5270092ea" path="/var/lib/kubelet/pods/d8e020b0-f83c-4c78-a7ea-aee5270092ea/volumes" Oct 31 03:50:17.312896 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:17.312863 4934 generic.go:356] "Generic (PLEG): container finished" podID="d8e020b0-f83c-4c78-a7ea-aee5270092ea" containerID="4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd" exitCode=137 Oct 31 03:50:17.313086 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:17.312902 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-fd3fa5" Oct 31 03:50:17.313086 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:17.312966 4934 scope.go:122] "RemoveContainer" containerID="4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd" Oct 31 03:50:17.322611 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:17.322592 4934 scope.go:122] "RemoveContainer" containerID="4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd" Oct 31 03:50:17.322934 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:17.322917 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd\": container with ID starting with 4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd not found: ID does not exist" containerID="4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd" Oct 31 03:50:17.323011 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:17.322943 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd"} err="failed to get container status \"4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd\": rpc error: code = NotFound desc = could not find container \"4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd\": container with ID starting with 4ba26c0e5c6d90b430092bac97fc2a7f7c506fcd62c187c6a0bd1fd19eb805cd not found: ID does not exist" Oct 31 03:50:19.167787 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:19.167750 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:50:28.848807 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.848775 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp"] Oct 31 03:50:28.849471 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.849451 4934 cpu_manager.go:414] "containerMap: removing container" podUID="d8e020b0-f83c-4c78-a7ea-aee5270092ea" containerName="test-curl-7153-fd3fa5" Oct 31 03:50:28.849559 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.849471 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e020b0-f83c-4c78-a7ea-aee5270092ea" containerName="test-curl-7153-fd3fa5" Oct 31 03:50:28.849660 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.849646 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e020b0-f83c-4c78-a7ea-aee5270092ea" containerName="test-curl-7153-fd3fa5" Oct 31 03:50:28.854122 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.854101 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:28.931758 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.931729 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9933ee-1180-4228-a595-440a790a7d20-openshift-service-ca\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:28.931887 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.931793 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjflv\" (UniqueName: \"kubernetes.io/projected/4c9933ee-1180-4228-a595-440a790a7d20-kube-api-access-xjflv\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:28.932054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.932026 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:28.932123 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.932074 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-metrics-certs\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:28.958765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.958742 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7c43a6-serving-cert\"" Oct 31 03:50:28.961442 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:28.961422 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp"] Oct 31 03:50:29.033102 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.033067 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9933ee-1180-4228-a595-440a790a7d20-openshift-service-ca\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.033240 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.033143 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjflv\" (UniqueName: \"kubernetes.io/projected/4c9933ee-1180-4228-a595-440a790a7d20-kube-api-access-xjflv\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.033320 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.033286 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.033379 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.033321 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-metrics-certs\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.033427 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:29.033407 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-serving-cert: secret "maas-api-e2e-api-a-7c43a6-serving-cert" not found Oct 31 03:50:29.033472 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:29.033462 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls podName:4c9933ee-1180-4228-a595-440a790a7d20 nodeName:}" failed. No retries permitted until 2026-10-31 03:50:29.533447117 +0000 UTC m=+1862.852189722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls") pod "maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" (UID: "4c9933ee-1180-4228-a595-440a790a7d20") : secret "maas-api-e2e-api-a-7c43a6-serving-cert" not found Oct 31 03:50:29.033654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.033634 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9933ee-1180-4228-a595-440a790a7d20-openshift-service-ca\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.035809 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.035790 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-metrics-certs\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.094555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.094537 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjflv\" (UniqueName: \"kubernetes.io/projected/4c9933ee-1180-4228-a595-440a790a7d20-kube-api-access-xjflv\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.539460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.539427 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.541942 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.541918 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls\") pod \"maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:29.766343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:29.766310 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:30.150873 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.150846 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp"] Oct 31 03:50:30.152195 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:50:30.152169 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9933ee_1180_4228_a595_440a790a7d20.slice/crio-cef98e05ea01aba04d8f27e8411ac27faddf24f328dc87fb7e17e6b478731000 WatchSource:0}: Error finding container cef98e05ea01aba04d8f27e8411ac27faddf24f328dc87fb7e17e6b478731000: Status 404 returned error can't find the container with id cef98e05ea01aba04d8f27e8411ac27faddf24f328dc87fb7e17e6b478731000 Oct 31 03:50:30.200753 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.200723 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4"] Oct 31 03:50:30.205563 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.205545 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.247936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.247900 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.247944 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248118 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.248099 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248180 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.248164 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248320 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.248300 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlglj\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-kube-api-access-hlglj\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248371 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.248358 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.248441 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.248469 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.248587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.248570 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.258118 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.258086 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-096812-openshift-default-dockercfg-7vgck\"" Oct 31 03:50:30.266153 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.266112 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4"] Oct 31 03:50:30.349972 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.349946 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350086 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.349991 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350086 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350024 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350083 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350165 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350337 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350265 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlglj\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-kube-api-access-hlglj\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350337 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350317 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350433 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350381 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350433 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350410 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350548 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350510 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350618 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350555 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350748 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350725 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.350835 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.350751 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.351074 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.351054 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.352416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.352396 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.352596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.352581 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.403348 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.403274 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" event={"ID":"4c9933ee-1180-4228-a595-440a790a7d20","Type":"ContainerStarted","Data":"cef98e05ea01aba04d8f27e8411ac27faddf24f328dc87fb7e17e6b478731000"} Oct 31 03:50:30.411156 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.411133 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.411330 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.411314 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlglj\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-kube-api-access-hlglj\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.521885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.521856 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:30.970324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.970287 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4"] Oct 31 03:50:30.979765 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:50:30.979736 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272e4f32_c533_4267_ba31_b2b94ec7e5dd.slice/crio-f97bb4bbf376a691790269a08c478d68c4e00df688d8c4ccda805a9c34fee009 WatchSource:0}: Error finding container f97bb4bbf376a691790269a08c478d68c4e00df688d8c4ccda805a9c34fee009: Status 404 returned error can't find the container with id f97bb4bbf376a691790269a08c478d68c4e00df688d8c4ccda805a9c34fee009 Oct 31 03:50:30.990279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.988120 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:30.990279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.988276 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:30.990279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:30.988345 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:31.413143 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:31.413093 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" event={"ID":"272e4f32-c533-4267-ba31-b2b94ec7e5dd","Type":"ContainerStarted","Data":"aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa"} Oct 31 03:50:31.413631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:31.413171 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" event={"ID":"272e4f32-c533-4267-ba31-b2b94ec7e5dd","Type":"ContainerStarted","Data":"f97bb4bbf376a691790269a08c478d68c4e00df688d8c4ccda805a9c34fee009"} Oct 31 03:50:31.414895 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:31.414870 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" event={"ID":"4c9933ee-1180-4228-a595-440a790a7d20","Type":"ContainerStarted","Data":"9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907"} Oct 31 03:50:31.415093 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:31.415072 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:31.521931 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:31.521901 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:31.535982 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:31.535930 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podStartSLOduration=1.535918005 podStartE2EDuration="1.535918005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:31.533573547 +0000 UTC m=+1864.852316185" watchObservedRunningTime="2026-10-31 03:50:31.535918005 +0000 UTC m=+1864.854660717" Oct 31 03:50:31.652552 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:31.652504 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" podStartSLOduration=2.790661998 podStartE2EDuration="3.652489553s" totalImagesPullingTime="861.827555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:31.651696862 +0000 UTC m=+1864.970439492" watchObservedRunningTime="2026-10-31 03:50:31.652489553 +0000 UTC m=+1864.971232179" Oct 31 03:50:32.524646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:32.524616 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:50:32.525048 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:32.524677 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 03:50:33.522969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:33.522922 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:50:33.523132 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:33.522995 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 03:50:34.522700 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:34.522667 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:50:34.523070 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:34.522724 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 03:50:35.530252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:35.524689 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:50:35.530252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:35.524751 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 03:50:36.522173 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:36.522122 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:50:36.522354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:36.522190 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 03:50:37.428119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:37.428091 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:50:37.522581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:37.522549 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:50:37.522743 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:37.522593 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 03:50:38.657631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:38.657600 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:38.658863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:38.658841 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:38.660764 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:38.660736 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:50:44.886431 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:44.886393 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5"] Oct 31 03:50:44.886841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:44.886627 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" containerID="cri-o://3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef" gracePeriod=30 Oct 31 03:50:46.100924 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.100888 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q"] Oct 31 03:50:46.101408 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.101081 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" podUID="5f26a324-66d1-4a3a-a883-37d28fd726a1" containerName="discovery" containerID="cri-o://18389bfc1402608f14b5c9bfebf7f0cf8d5fdb054526987d71817136a0f45564" gracePeriod=30 Oct 31 03:50:46.151372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.151343 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f"] Oct 31 03:50:46.151562 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.151541 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" podUID="aaeba949-b325-4687-ad25-c03f4a0d14a8" containerName="discovery" containerID="cri-o://9f541eda6b237c8b272cb130d7c874f698ce41b8efd71fabfddea4caf1da85b3" gracePeriod=30 Oct 31 03:50:46.573543 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.573496 4934 generic.go:356] "Generic (PLEG): container finished" podID="5f26a324-66d1-4a3a-a883-37d28fd726a1" containerID="18389bfc1402608f14b5c9bfebf7f0cf8d5fdb054526987d71817136a0f45564" exitCode=0 Oct 31 03:50:46.573681 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.573627 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" event={"ID":"5f26a324-66d1-4a3a-a883-37d28fd726a1","Type":"ContainerDied","Data":"18389bfc1402608f14b5c9bfebf7f0cf8d5fdb054526987d71817136a0f45564"} Oct 31 03:50:46.601653 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.577082 4934 generic.go:356] "Generic (PLEG): container finished" podID="aaeba949-b325-4687-ad25-c03f4a0d14a8" containerID="9f541eda6b237c8b272cb130d7c874f698ce41b8efd71fabfddea4caf1da85b3" exitCode=0 Oct 31 03:50:46.601653 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.577167 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" event={"ID":"aaeba949-b325-4687-ad25-c03f4a0d14a8","Type":"ContainerDied","Data":"9f541eda6b237c8b272cb130d7c874f698ce41b8efd71fabfddea4caf1da85b3"} Oct 31 03:50:46.601653 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.584091 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:50:46.647597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.647567 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:50:46.677795 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.677711 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-kubeconfig\") pod \"aaeba949-b325-4687-ad25-c03f4a0d14a8\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " Oct 31 03:50:46.677795 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.677766 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-token\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-token\") pod \"aaeba949-b325-4687-ad25-c03f4a0d14a8\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " Oct 31 03:50:46.678015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.677809 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-dns-cert\") pod \"aaeba949-b325-4687-ad25-c03f4a0d14a8\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " Oct 31 03:50:46.678015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.677901 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-kube-api-access-bpv6z\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-kube-api-access-bpv6z\") pod \"aaeba949-b325-4687-ad25-c03f4a0d14a8\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " Oct 31 03:50:46.678161 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.678045 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-cacerts\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-cacerts\") pod \"aaeba949-b325-4687-ad25-c03f4a0d14a8\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " Oct 31 03:50:46.678161 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.678110 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-ca-configmap\") pod \"aaeba949-b325-4687-ad25-c03f4a0d14a8\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " Oct 31 03:50:46.678288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.678275 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaeba949-b325-4687-ad25-c03f4a0d14a8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaeba949-b325-4687-ad25-c03f4a0d14a8-local-certs\") pod \"aaeba949-b325-4687-ad25-c03f4a0d14a8\" (UID: \"aaeba949-b325-4687-ad25-c03f4a0d14a8\") " Oct 31 03:50:46.679093 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.679058 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-ca-configmap" pod "aaeba949-b325-4687-ad25-c03f4a0d14a8" (UID: "aaeba949-b325-4687-ad25-c03f4a0d14a8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:46.681024 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.680996 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-cacerts" pod "aaeba949-b325-4687-ad25-c03f4a0d14a8" (UID: "aaeba949-b325-4687-ad25-c03f4a0d14a8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:46.681024 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.681006 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-dns-cert" pod "aaeba949-b325-4687-ad25-c03f4a0d14a8" (UID: "aaeba949-b325-4687-ad25-c03f4a0d14a8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:46.681248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.681111 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-kubeconfig" pod "aaeba949-b325-4687-ad25-c03f4a0d14a8" (UID: "aaeba949-b325-4687-ad25-c03f4a0d14a8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:46.681330 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.681278 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-token" pod "aaeba949-b325-4687-ad25-c03f4a0d14a8" (UID: "aaeba949-b325-4687-ad25-c03f4a0d14a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:46.681449 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.681430 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaeba949-b325-4687-ad25-c03f4a0d14a8-local-certs" pod "aaeba949-b325-4687-ad25-c03f4a0d14a8" (UID: "aaeba949-b325-4687-ad25-c03f4a0d14a8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:46.682438 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.682413 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-kube-api-access-bpv6z" pod "aaeba949-b325-4687-ad25-c03f4a0d14a8" (UID: "aaeba949-b325-4687-ad25-c03f4a0d14a8"). InnerVolumeSpecName "kube-api-access-bpv6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:46.779263 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.779219 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-dns-cert\") pod \"5f26a324-66d1-4a3a-a883-37d28fd726a1\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " Oct 31 03:50:46.779444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.779297 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f26a324-66d1-4a3a-a883-37d28fd726a1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26a324-66d1-4a3a-a883-37d28fd726a1-local-certs\") pod \"5f26a324-66d1-4a3a-a883-37d28fd726a1\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " Oct 31 03:50:46.779444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.779344 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-kubeconfig\") pod \"5f26a324-66d1-4a3a-a883-37d28fd726a1\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " Oct 31 03:50:46.779444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.779387 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-cacerts\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-cacerts\") pod \"5f26a324-66d1-4a3a-a883-37d28fd726a1\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " Oct 31 03:50:46.779609 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.779445 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-token\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-token\") pod \"5f26a324-66d1-4a3a-a883-37d28fd726a1\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " Oct 31 03:50:46.779609 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.779476 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-ca-configmap\") pod \"5f26a324-66d1-4a3a-a883-37d28fd726a1\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " Oct 31 03:50:46.779722 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.779654 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-kube-api-access-p9z4c\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-kube-api-access-p9z4c\") pod \"5f26a324-66d1-4a3a-a883-37d28fd726a1\" (UID: \"5f26a324-66d1-4a3a-a883-37d28fd726a1\") " Oct 31 03:50:46.780147 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780075 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-ca-configmap" pod "5f26a324-66d1-4a3a-a883-37d28fd726a1" (UID: "5f26a324-66d1-4a3a-a883-37d28fd726a1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:46.780744 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780720 4934 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-cacerts\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.780832 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780749 4934 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-ca-configmap\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.780832 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780766 4934 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaeba949-b325-4687-ad25-c03f4a0d14a8-local-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.780832 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780781 4934 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-kubeconfig\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.780832 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780797 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.780832 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780811 4934 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-ca-configmap\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.780832 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780825 4934 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aaeba949-b325-4687-ad25-c03f4a0d14a8-istio-csr-dns-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.781133 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.780838 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpv6z\" (UniqueName: \"kubernetes.io/projected/aaeba949-b325-4687-ad25-c03f4a0d14a8-kube-api-access-bpv6z\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.782311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.782283 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-kubeconfig" pod "5f26a324-66d1-4a3a-a883-37d28fd726a1" (UID: "5f26a324-66d1-4a3a-a883-37d28fd726a1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:46.782311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.782291 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f26a324-66d1-4a3a-a883-37d28fd726a1-local-certs" pod "5f26a324-66d1-4a3a-a883-37d28fd726a1" (UID: "5f26a324-66d1-4a3a-a883-37d28fd726a1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:46.782443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.782377 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-dns-cert" pod "5f26a324-66d1-4a3a-a883-37d28fd726a1" (UID: "5f26a324-66d1-4a3a-a883-37d28fd726a1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:46.782660 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.782640 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-cacerts" pod "5f26a324-66d1-4a3a-a883-37d28fd726a1" (UID: "5f26a324-66d1-4a3a-a883-37d28fd726a1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:46.782821 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.782806 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-token" pod "5f26a324-66d1-4a3a-a883-37d28fd726a1" (UID: "5f26a324-66d1-4a3a-a883-37d28fd726a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:46.783029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.783015 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-kube-api-access-p9z4c" pod "5f26a324-66d1-4a3a-a883-37d28fd726a1" (UID: "5f26a324-66d1-4a3a-a883-37d28fd726a1"). InnerVolumeSpecName "kube-api-access-p9z4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:46.925284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.924321 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9z4c\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-kube-api-access-p9z4c\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.925284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.924359 4934 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-csr-dns-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.925284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.924373 4934 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26a324-66d1-4a3a-a883-37d28fd726a1-local-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.925284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.924390 4934 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-kubeconfig\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.925284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.924403 4934 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f26a324-66d1-4a3a-a883-37d28fd726a1-cacerts\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:46.925284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:46.924418 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26a324-66d1-4a3a-a883-37d28fd726a1-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:47.583480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.583454 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" Oct 31 03:50:47.583480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.583463 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q" event={"ID":"5f26a324-66d1-4a3a-a883-37d28fd726a1","Type":"ContainerDied","Data":"9eaa95cd1efee0defae4fc2cb711b504a4fe53e66b058e42653441d25331b7b5"} Oct 31 03:50:47.583998 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.583550 4934 scope.go:122] "RemoveContainer" containerID="18389bfc1402608f14b5c9bfebf7f0cf8d5fdb054526987d71817136a0f45564" Oct 31 03:50:47.584764 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.584740 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" event={"ID":"aaeba949-b325-4687-ad25-c03f4a0d14a8","Type":"ContainerDied","Data":"117a626e498da17ce7e9f5894a7a95f334b903b7fc6b2ecf245346b4b17a2f23"} Oct 31 03:50:47.584885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.584765 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f" Oct 31 03:50:47.595646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.595630 4934 scope.go:122] "RemoveContainer" containerID="9f541eda6b237c8b272cb130d7c874f698ce41b8efd71fabfddea4caf1da85b3" Oct 31 03:50:47.750765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.750739 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q"] Oct 31 03:50:47.800725 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.800697 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trq9q"] Oct 31 03:50:47.894309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.893811 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f"] Oct 31 03:50:47.962252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:47.947806 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rrq6f"] Oct 31 03:50:48.823071 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:48.823035 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f26a324-66d1-4a3a-a883-37d28fd726a1" path="/var/lib/kubelet/pods/5f26a324-66d1-4a3a-a883-37d28fd726a1/volumes" Oct 31 03:50:48.823556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:48.823541 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaeba949-b325-4687-ad25-c03f4a0d14a8" path="/var/lib/kubelet/pods/aaeba949-b325-4687-ad25-c03f4a0d14a8/volumes" Oct 31 03:50:49.163026 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:49.162951 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:50:50.147687 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.147659 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:50:50.279934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.279867 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-podinfo\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280053 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.279934 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-data\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280053 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.279967 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-socket\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280053 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280024 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-credential-socket\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280092 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-envoy\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280152 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-certs\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280180 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-kube-api-access-6dvkw\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-kube-api-access-6dvkw\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280407 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280259 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-token\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-token\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280407 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280307 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dbcd78-cf28-4850-935c-9ab453f1232f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dbcd78-cf28-4850-935c-9ab453f1232f-istiod-ca-cert\") pod \"97dbcd78-cf28-4850-935c-9ab453f1232f\" (UID: \"97dbcd78-cf28-4850-935c-9ab453f1232f\") " Oct 31 03:50:50.280407 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280371 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-credential-socket" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:50.280407 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280386 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-socket" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:50.280790 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280721 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-certs" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:50.280880 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280858 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.280945 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280884 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.280945 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280887 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dbcd78-cf28-4850-935c-9ab453f1232f-istiod-ca-cert" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:50.280945 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280900 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.281102 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.280946 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-data" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:50.282280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.282252 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-podinfo" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:50.282556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.282538 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-token" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:50.282645 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.282603 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-envoy" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:50.282768 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.282753 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-kube-api-access-6dvkw" pod "97dbcd78-cf28-4850-935c-9ab453f1232f" (UID: "97dbcd78-cf28-4850-935c-9ab453f1232f"). InnerVolumeSpecName "kube-api-access-6dvkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:50.382087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.382057 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.382087 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.382084 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.382349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.382101 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.382349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.382110 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dvkw\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-kube-api-access-6dvkw\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.382349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.382122 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dbcd78-cf28-4850-935c-9ab453f1232f-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.382349 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.382153 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dbcd78-cf28-4850-935c-9ab453f1232f-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:50.606034 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.605996 4934 generic.go:356] "Generic (PLEG): container finished" podID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerID="3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef" exitCode=0 Oct 31 03:50:50.606221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.606041 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" Oct 31 03:50:50.606221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.606068 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" event={"ID":"97dbcd78-cf28-4850-935c-9ab453f1232f","Type":"ContainerDied","Data":"3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef"} Oct 31 03:50:50.606221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.606125 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5" event={"ID":"97dbcd78-cf28-4850-935c-9ab453f1232f","Type":"ContainerDied","Data":"1017f8e6d0cc30ab8538257ea7b35171e01301428e56cee4ee4f244a11b04b12"} Oct 31 03:50:50.606221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.606166 4934 scope.go:122] "RemoveContainer" containerID="3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef" Oct 31 03:50:50.616001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.615941 4934 scope.go:122] "RemoveContainer" containerID="3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef" Oct 31 03:50:50.616263 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:50:50.616214 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef\": container with ID starting with 3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef not found: ID does not exist" containerID="3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef" Oct 31 03:50:50.616334 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.616276 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef"} err="failed to get container status \"3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef\": rpc error: code = NotFound desc = could not find container \"3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef\": container with ID starting with 3155abe710fcaf6e666d78a9b7295250346cb499f93f8cb56e397f043be824ef not found: ID does not exist" Oct 31 03:50:50.772538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.772513 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5"] Oct 31 03:50:50.834457 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:50.834419 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-d7bv5"] Oct 31 03:50:52.823403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:52.823370 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" path="/var/lib/kubelet/pods/97dbcd78-cf28-4850-935c-9ab453f1232f/volumes" Oct 31 03:50:55.818254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818207 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj"] Oct 31 03:50:55.818704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818691 4934 cpu_manager.go:414] "containerMap: removing container" podUID="aaeba949-b325-4687-ad25-c03f4a0d14a8" containerName="discovery" Oct 31 03:50:55.818748 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818705 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaeba949-b325-4687-ad25-c03f4a0d14a8" containerName="discovery" Oct 31 03:50:55.818748 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818717 4934 cpu_manager.go:414] "containerMap: removing container" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" Oct 31 03:50:55.818748 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818722 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" Oct 31 03:50:55.818843 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818755 4934 cpu_manager.go:414] "containerMap: removing container" podUID="5f26a324-66d1-4a3a-a883-37d28fd726a1" containerName="discovery" Oct 31 03:50:55.818843 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818760 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f26a324-66d1-4a3a-a883-37d28fd726a1" containerName="discovery" Oct 31 03:50:55.818843 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818839 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f26a324-66d1-4a3a-a883-37d28fd726a1" containerName="discovery" Oct 31 03:50:55.818927 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818849 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaeba949-b325-4687-ad25-c03f4a0d14a8" containerName="discovery" Oct 31 03:50:55.818927 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.818856 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dbcd78-cf28-4850-935c-9ab453f1232f" containerName="istio-proxy" Oct 31 03:50:55.823322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.823302 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.859689 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.859671 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-data\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.859798 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.859711 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-envoy\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.859854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.859821 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czslf\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-kube-api-access-czslf\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.859897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.859854 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-podinfo\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.859897 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.859883 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-token\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.860035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.860012 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-credential-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.860101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.860066 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520f796a-3221-4eaf-b27f-0b8c643476f1-istiod-ca-cert\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.860140 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.860106 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-certs\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.860186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.860175 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.875854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.875833 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7c43a6-openshift-default-dockercfg-dpqgw\"" Oct 31 03:50:55.929382 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.929355 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj"] Oct 31 03:50:55.961723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.961695 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.961846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.961775 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-data\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.961846 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.961818 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-envoy\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.961976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.961888 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czslf\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-kube-api-access-czslf\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.961976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.961931 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-podinfo\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.962098 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.961978 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-token\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.962098 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.962067 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-credential-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.962268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.962156 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520f796a-3221-4eaf-b27f-0b8c643476f1-istiod-ca-cert\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.962268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.962163 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.962268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.962190 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-certs\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.962604 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.962581 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-certs\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.963130 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.963107 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-credential-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.963736 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.963709 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520f796a-3221-4eaf-b27f-0b8c643476f1-istiod-ca-cert\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.964003 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.963980 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-data\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.966454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.966428 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-podinfo\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:55.970504 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:55.970479 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-envoy\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:56.023429 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.023409 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-token\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:56.023713 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.023692 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czslf\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-kube-api-access-czslf\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:56.135642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.135589 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:56.567676 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.567648 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj"] Oct 31 03:50:56.570078 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:50:56.570047 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520f796a_3221_4eaf_b27f_0b8c643476f1.slice/crio-9fca959504a3422e42f63abd26d6aadfcef4b273c2c0d4f1dd7a5e14d20ac571 WatchSource:0}: Error finding container 9fca959504a3422e42f63abd26d6aadfcef4b273c2c0d4f1dd7a5e14d20ac571: Status 404 returned error can't find the container with id 9fca959504a3422e42f63abd26d6aadfcef4b273c2c0d4f1dd7a5e14d20ac571 Oct 31 03:50:56.572631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.572599 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:56.572721 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.572662 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:56.572721 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.572689 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:50:56.642790 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:56.642762 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" event={"ID":"520f796a-3221-4eaf-b27f-0b8c643476f1","Type":"ContainerStarted","Data":"9fca959504a3422e42f63abd26d6aadfcef4b273c2c0d4f1dd7a5e14d20ac571"} Oct 31 03:50:57.649371 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:57.649334 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" event={"ID":"520f796a-3221-4eaf-b27f-0b8c643476f1","Type":"ContainerStarted","Data":"e295677dcfb62cc253cebb6548202b20369a6ee45bad82ce52a95e3c6d61ce5d"} Oct 31 03:50:57.817594 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:57.817530 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podStartSLOduration=2.817510476 podStartE2EDuration="2.817510476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:57.81543381 +0000 UTC m=+1891.134176438" watchObservedRunningTime="2026-10-31 03:50:57.817510476 +0000 UTC m=+1891.136253107" Oct 31 03:50:58.136637 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:58.136602 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:50:58.138278 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:58.138251 4934 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:50:58.138391 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:58.138302 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:50:59.136119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:59.136086 4934 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:50:59.136507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:50:59.136134 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:51:00.136312 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:00.136284 4934 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:51:00.136757 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:00.136735 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:51:01.136064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:01.136033 4934 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:51:01.136220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:01.136099 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:51:02.136593 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:02.136560 4934 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:51:02.139121 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:02.136620 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:51:03.141123 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:03.141098 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:51:03.141546 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:03.141441 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:51:03.142035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:03.142013 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:51:09.405815 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.405782 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs"] Oct 31 03:51:09.410029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.410007 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.462414 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.462388 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-1783b3-serving-cert\"" Oct 31 03:51:09.515150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.515122 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs"] Oct 31 03:51:09.539152 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.539110 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6t7\" (UniqueName: \"kubernetes.io/projected/f05f22f9-4f34-4f85-836b-6d895a800855-kube-api-access-nn6t7\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.539352 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.539331 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.539488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.539468 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.539563 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.539531 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f22f9-4f34-4f85-836b-6d895a800855-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.640960 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.640926 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.641112 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.641008 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.641112 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.641054 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f22f9-4f34-4f85-836b-6d895a800855-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.641112 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:51:09.641080 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-serving-cert: secret "maas-api-e2e-shared-b-w6-1783b3-serving-cert" not found Oct 31 03:51:09.641302 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:51:09.641140 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls podName:f05f22f9-4f34-4f85-836b-6d895a800855 nodeName:}" failed. No retries permitted until 2026-10-31 03:51:10.141121992 +0000 UTC m=+1903.459864620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls") pod "maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" (UID: "f05f22f9-4f34-4f85-836b-6d895a800855") : secret "maas-api-e2e-shared-b-w6-1783b3-serving-cert" not found Oct 31 03:51:09.641302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.641173 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6t7\" (UniqueName: \"kubernetes.io/projected/f05f22f9-4f34-4f85-836b-6d895a800855-kube-api-access-nn6t7\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.641774 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.641752 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f22f9-4f34-4f85-836b-6d895a800855-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.643514 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.643497 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:09.703141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:09.703061 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6t7\" (UniqueName: \"kubernetes.io/projected/f05f22f9-4f34-4f85-836b-6d895a800855-kube-api-access-nn6t7\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:10.146787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:10.146757 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:10.149158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:10.149132 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:10.324728 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:10.324696 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:10.702213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:10.702122 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs"] Oct 31 03:51:10.725803 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:51:10.725066 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05f22f9_4f34_4f85_836b_6d895a800855.slice/crio-05c69a3355f159b62cfc11be16360054c9bece1c420e857133ba93d0979ed6a2 WatchSource:0}: Error finding container 05c69a3355f159b62cfc11be16360054c9bece1c420e857133ba93d0979ed6a2: Status 404 returned error can't find the container with id 05c69a3355f159b62cfc11be16360054c9bece1c420e857133ba93d0979ed6a2 Oct 31 03:51:10.768147 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:10.768114 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" event={"ID":"f05f22f9-4f34-4f85-836b-6d895a800855","Type":"ContainerStarted","Data":"05c69a3355f159b62cfc11be16360054c9bece1c420e857133ba93d0979ed6a2"} Oct 31 03:51:11.775969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:11.775934 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" event={"ID":"f05f22f9-4f34-4f85-836b-6d895a800855","Type":"ContainerStarted","Data":"21f5269d7f08d31074ef20cd782642d5638ca269fda02a1cd528033dcd082703"} Oct 31 03:51:11.776434 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:11.776018 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:11.899602 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:11.899505 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" podStartSLOduration=2.023006812 podStartE2EDuration="2.899491942s" totalImagesPullingTime="876.48513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:11.898647442 +0000 UTC m=+1905.217390070" watchObservedRunningTime="2026-10-31 03:51:11.899491942 +0000 UTC m=+1905.218234568" Oct 31 03:51:17.787269 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:17.787220 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:51:19.165265 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:51:19.165204 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:51:27.930903 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:27.930813 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj"] Oct 31 03:51:27.936262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:27.936240 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:27.990589 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:27.989593 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e02dd5-openshift-default-dockercfg-6sg5q\"" Oct 31 03:51:28.039032 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.038988 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-credential-socket\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.041591 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.041562 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-socket\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.041853 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.041834 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-envoy\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.042117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.042093 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istiod-ca-cert\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.042361 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.042344 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-podinfo\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.042602 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.042582 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-data\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.042764 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.042748 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-token\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.042972 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.042953 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhkd\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-kube-api-access-dkhkd\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.043122 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.043107 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-certs\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.090248 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.076280 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj"] Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170190 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhkd\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-kube-api-access-dkhkd\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170257 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-certs\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170304 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-credential-socket\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170363 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-socket\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170388 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-envoy\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170483 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istiod-ca-cert\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170530 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-podinfo\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170617 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-data\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.175598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.170650 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-token\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.209367 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.208696 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-socket\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.209541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.209445 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-certs\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.212345 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.210667 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istiod-ca-cert\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.224120 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.224095 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-data\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.237965 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.232953 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-credential-socket\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.237965 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.236917 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-podinfo\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.238794 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.238766 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-envoy\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.239581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.239559 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-token\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.269944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.269916 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhkd\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-kube-api-access-dkhkd\") pod \"e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.553499 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.553415 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:28.998109 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:28.998062 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj"] Oct 31 03:51:29.030066 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:51:29.030039 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd106d91d_2fbd_4dbb_8b6f_5601d33d95b0.slice/crio-5fc99da8f68e6df22f5e037d4f7734529747db45bb663302538ff2a7621ac06d WatchSource:0}: Error finding container 5fc99da8f68e6df22f5e037d4f7734529747db45bb663302538ff2a7621ac06d: Status 404 returned error can't find the container with id 5fc99da8f68e6df22f5e037d4f7734529747db45bb663302538ff2a7621ac06d Oct 31 03:51:29.032603 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:29.032568 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:29.032723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:29.032633 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:29.032723 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:29.032660 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:29.904324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:29.904248 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" event={"ID":"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0","Type":"ContainerStarted","Data":"3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64"} Oct 31 03:51:29.904626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:29.904603 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" event={"ID":"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0","Type":"ContainerStarted","Data":"5fc99da8f68e6df22f5e037d4f7734529747db45bb663302538ff2a7621ac06d"} Oct 31 03:51:30.085272 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.085212 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podStartSLOduration=3.085198556 podStartE2EDuration="3.085198556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:30.083477435 +0000 UTC m=+1923.402220066" watchObservedRunningTime="2026-10-31 03:51:30.085198556 +0000 UTC m=+1923.403941183" Oct 31 03:51:30.554468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.554429 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:30.555282 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.555255 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm"] Oct 31 03:51:30.559400 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.559369 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.559755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.559730 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:30.635255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.632372 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-84e130-serving-cert\"" Oct 31 03:51:30.635255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.634664 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm"] Oct 31 03:51:30.738262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.737886 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-metrics-certs\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.738262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.737960 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwbk\" (UniqueName: \"kubernetes.io/projected/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-kube-api-access-hbwbk\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.738262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.737986 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-openshift-service-ca\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.738262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.738028 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-maas-api-tls\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.840315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.840281 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-maas-api-tls\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.841124 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.841091 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-metrics-certs\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.841289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.841192 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwbk\" (UniqueName: \"kubernetes.io/projected/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-kube-api-access-hbwbk\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.841289 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.841255 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-openshift-service-ca\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.843858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.842988 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-openshift-service-ca\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.848350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.848316 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-maas-api-tls\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.850015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.849990 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-metrics-certs\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.903533 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.903509 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwbk\" (UniqueName: \"kubernetes.io/projected/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-kube-api-access-hbwbk\") pod \"maas-api-e2e-api-a-84e130-74d5c87848-7kzmm\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:30.911384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.911354 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:30.912442 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:30.912422 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:51:31.175506 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:31.175426 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:31.615252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:31.596350 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm"] Oct 31 03:51:31.918396 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:31.918292 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" event={"ID":"ac2d9504-ef57-4231-bfa3-67ae4a8db56e","Type":"ContainerStarted","Data":"87a008628f162bafe95a0d5c6a884f48cc4071c6c82260745d6a5b0e0f258f8a"} Oct 31 03:51:32.928311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:32.928262 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" event={"ID":"ac2d9504-ef57-4231-bfa3-67ae4a8db56e","Type":"ContainerStarted","Data":"be4eba77266fab017c9ac0ab9dbf422bfc15c79f00ca641fc0da1445dcef83f4"} Oct 31 03:51:32.928854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:32.928755 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:33.050173 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:33.050084 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" podStartSLOduration=2.198361945 podStartE2EDuration="3.050070354s" totalImagesPullingTime="851.708409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:33.048457918 +0000 UTC m=+1926.367200544" watchObservedRunningTime="2026-10-31 03:51:33.050070354 +0000 UTC m=+1926.368812981" Oct 31 03:51:34.051037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:34.050996 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4"] Oct 31 03:51:34.051551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:34.051274 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" containerID="cri-o://aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa" gracePeriod=30 Oct 31 03:51:38.477326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.477250 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57"] Oct 31 03:51:38.501268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.483261 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519172 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519144 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519321 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519288 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519321 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519314 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519384 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519431 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519419 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519479 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519524 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519515 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519633 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519614 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.519670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.519651 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-kube-api-access-7svf4\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.525922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.525900 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh"] Oct 31 03:51:38.529603 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.529587 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.572829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.572808 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-1783b3-openshift-default-dockercfg-2fskz\"" Oct 31 03:51:38.575524 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.575500 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57"] Oct 31 03:51:38.593079 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.593055 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh"] Oct 31 03:51:38.620998 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.620971 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621125 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621006 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621125 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621025 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621155 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621313 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621216 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a111075-2227-42a4-a864-54fe1fbc54a6-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621313 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621286 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621354 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621417 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621390 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621447 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621504 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621510 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621553 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npg8q\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-kube-api-access-npg8q\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621623 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621671 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621707 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621761 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621789 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.621913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621819 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621830 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-kube-api-access-7svf4\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.621913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.621861 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.622302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.622281 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.622822 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.622735 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.623326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.623304 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.624307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.624281 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.624765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.624742 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.661288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.661265 4934 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:51:38.661493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.661319 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:51:38.683051 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.683025 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-kube-api-access-7svf4\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.683146 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.683112 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.723050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723027 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723077 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723111 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723156 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723189 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723368 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723251 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723368 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723309 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a111075-2227-42a4-a864-54fe1fbc54a6-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723368 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723338 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723453 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npg8q\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-kube-api-access-npg8q\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723618 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723596 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723685 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723617 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723685 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723674 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.723842 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.723826 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.724538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.724516 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a111075-2227-42a4-a864-54fe1fbc54a6-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.725466 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.725447 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.725536 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.725483 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.784710 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.784648 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.784798 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.784727 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npg8q\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-kube-api-access-npg8q\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.805302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.805284 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:38.841807 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.841787 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:38.944392 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:38.944341 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:51:39.239022 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.238990 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57"] Oct 31 03:51:39.248852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.248819 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:39.248969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.248899 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:39.248969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.248945 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:39.307158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.306933 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh"] Oct 31 03:51:39.314901 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:51:39.314830 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a111075_2227_42a4_a864_54fe1fbc54a6.slice/crio-eb5b3636607c1b0afad3772f11cef37a81dda490c7ee0fa734f0889013b28735 WatchSource:0}: Error finding container eb5b3636607c1b0afad3772f11cef37a81dda490c7ee0fa734f0889013b28735: Status 404 returned error can't find the container with id eb5b3636607c1b0afad3772f11cef37a81dda490c7ee0fa734f0889013b28735 Oct 31 03:51:39.318347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.317551 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:39.318347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.317641 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:39.318347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.317683 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:51:39.334266 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.334208 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:51:39.432724 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.432693 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-credential-socket\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.432863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.432773 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-certs\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.432863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.432821 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istiod-ca-cert\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.432983 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.432944 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-kube-api-access-hlglj\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-kube-api-access-hlglj\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.432983 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.432963 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-credential-socket" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.433082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433029 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-data\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.433132 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433092 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-envoy\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.433183 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433138 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-token\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-token\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.433259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433203 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-podinfo\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.433318 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433263 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-socket\") pod \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\" (UID: \"272e4f32-c533-4267-ba31-b2b94ec7e5dd\") " Oct 31 03:51:39.434006 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433836 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.434602 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433200 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-certs" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.434602 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433393 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-data" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.434602 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.433689 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istiod-ca-cert" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:51:39.434967 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.434934 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-socket" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.436454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.436420 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-envoy" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.437066 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.437040 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-token" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:39.437656 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.437636 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-podinfo" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:51:39.437737 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.437655 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-kube-api-access-hlglj" pod "272e4f32-c533-4267-ba31-b2b94ec7e5dd" (UID: "272e4f32-c533-4267-ba31-b2b94ec7e5dd"). InnerVolumeSpecName "kube-api-access-hlglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:39.535369 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535335 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.535369 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535368 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.535816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535380 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.535816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535393 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.535816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535407 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.535816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535419 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.535816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535432 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlglj\" (UniqueName: \"kubernetes.io/projected/272e4f32-c533-4267-ba31-b2b94ec7e5dd-kube-api-access-hlglj\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.535816 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.535444 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/272e4f32-c533-4267-ba31-b2b94ec7e5dd-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.978675 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.978584 4934 generic.go:356] "Generic (PLEG): container finished" podID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerID="aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa" exitCode=0 Oct 31 03:51:39.978908 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.978637 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" Oct 31 03:51:39.979046 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.978655 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" event={"ID":"272e4f32-c533-4267-ba31-b2b94ec7e5dd","Type":"ContainerDied","Data":"aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa"} Oct 31 03:51:39.979115 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.979088 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4" event={"ID":"272e4f32-c533-4267-ba31-b2b94ec7e5dd","Type":"ContainerDied","Data":"f97bb4bbf376a691790269a08c478d68c4e00df688d8c4ccda805a9c34fee009"} Oct 31 03:51:39.979171 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.979127 4934 scope.go:122] "RemoveContainer" containerID="aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa" Oct 31 03:51:39.983200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.982095 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" event={"ID":"e2375106-1e2f-44a8-a7bf-dda13216fe2a","Type":"ContainerStarted","Data":"6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6"} Oct 31 03:51:39.983200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.982173 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" event={"ID":"e2375106-1e2f-44a8-a7bf-dda13216fe2a","Type":"ContainerStarted","Data":"95a2d3980194874372f2dcfc7d56b42853962719581ba9ec15eef22ad50a7d09"} Oct 31 03:51:39.985555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.985515 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" event={"ID":"2a111075-2227-42a4-a864-54fe1fbc54a6","Type":"ContainerStarted","Data":"ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6"} Oct 31 03:51:39.985854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.985821 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" event={"ID":"2a111075-2227-42a4-a864-54fe1fbc54a6","Type":"ContainerStarted","Data":"eb5b3636607c1b0afad3772f11cef37a81dda490c7ee0fa734f0889013b28735"} Oct 31 03:51:39.996690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.996655 4934 scope.go:122] "RemoveContainer" containerID="aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa" Oct 31 03:51:39.997123 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:51:39.997095 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa\": container with ID starting with aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa not found: ID does not exist" containerID="aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa" Oct 31 03:51:39.997293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:39.997134 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa"} err="failed to get container status \"aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa\": rpc error: code = NotFound desc = could not find container \"aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa\": container with ID starting with aa2ab36f89f3bb922e2b6bb8ecf9908d5c60df43bd559158268e9ca39ae746aa not found: ID does not exist" Oct 31 03:51:40.145833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.145801 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4"] Oct 31 03:51:40.151995 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.151970 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-rxfv4"] Oct 31 03:51:40.276309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.274009 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" podStartSLOduration=2.273989103 podStartE2EDuration="2.273989103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:40.270681616 +0000 UTC m=+1933.589424246" watchObservedRunningTime="2026-10-31 03:51:40.273989103 +0000 UTC m=+1933.592731731" Oct 31 03:51:40.811256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.808413 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:40.811256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.810745 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:51:40.811256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.810814 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:51:40.825514 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.825483 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" path="/var/lib/kubelet/pods/272e4f32-c533-4267-ba31-b2b94ec7e5dd/volumes" Oct 31 03:51:40.842424 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.842398 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:40.843944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.843925 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:51:40.844041 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:40.843962 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:51:41.816332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:41.814621 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:51:41.816332 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:41.814680 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:51:41.842375 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:41.842334 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:51:41.842540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:41.842387 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:51:42.820254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:42.819467 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:51:42.820254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:42.819526 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:51:42.842565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:42.842534 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:51:42.842726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:42.842585 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:51:43.574255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.571729 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podStartSLOduration=5.571706729 podStartE2EDuration="5.571706729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:40.402540416 +0000 UTC m=+1933.721283045" watchObservedRunningTime="2026-10-31 03:51:43.571706729 +0000 UTC m=+1936.890449357" Oct 31 03:51:43.574255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.574193 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7165-17251b"] Oct 31 03:51:43.575259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.575117 4934 cpu_manager.go:414] "containerMap: removing container" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" Oct 31 03:51:43.575259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.575140 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" Oct 31 03:51:43.580264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.579430 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="272e4f32-c533-4267-ba31-b2b94ec7e5dd" containerName="istio-proxy" Oct 31 03:51:43.589255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.589043 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:43.627527 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.627490 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7165-17251b"] Oct 31 03:51:43.637892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.637728 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkfj7\" (UniqueName: \"kubernetes.io/projected/2148d4f1-2bfb-4de1-9646-94bea2440ff6-kube-api-access-qkfj7\") pod \"test-curl-7165-17251b\" (UID: \"2148d4f1-2bfb-4de1-9646-94bea2440ff6\") " pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:43.685513 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.685463 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hppm7\"" Oct 31 03:51:43.739622 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.739574 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkfj7\" (UniqueName: \"kubernetes.io/projected/2148d4f1-2bfb-4de1-9646-94bea2440ff6-kube-api-access-qkfj7\") pod \"test-curl-7165-17251b\" (UID: \"2148d4f1-2bfb-4de1-9646-94bea2440ff6\") " pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:43.816004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.815955 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:51:43.816210 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.816034 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:51:43.820718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.820687 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkfj7\" (UniqueName: \"kubernetes.io/projected/2148d4f1-2bfb-4de1-9646-94bea2440ff6-kube-api-access-qkfj7\") pod \"test-curl-7165-17251b\" (UID: \"2148d4f1-2bfb-4de1-9646-94bea2440ff6\") " pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:43.842630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.842562 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:51:43.842630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.842608 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:51:43.922471 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:43.922421 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:44.371555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:44.371507 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7165-17251b"] Oct 31 03:51:44.813250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:44.806506 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:51:44.813250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:44.806564 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:51:44.843103 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:44.843070 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:51:44.843495 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:44.843117 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:51:45.048108 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.048070 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7165-17251b" event={"ID":"2148d4f1-2bfb-4de1-9646-94bea2440ff6","Type":"ContainerStarted","Data":"df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089"} Oct 31 03:51:45.048108 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.048115 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7165-17251b" event={"ID":"2148d4f1-2bfb-4de1-9646-94bea2440ff6","Type":"ContainerStarted","Data":"08b0d3bd9ee833222a701a2561924fcf46abd53f7739e3c0ac86ce27d6d09aa5"} Oct 31 03:51:45.172313 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.171761 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7165-17251b" podStartSLOduration=2.171740957 podStartE2EDuration="2.171740957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:45.170375623 +0000 UTC m=+1938.489118252" watchObservedRunningTime="2026-10-31 03:51:45.171740957 +0000 UTC m=+1938.490483584" Oct 31 03:51:45.859667 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.859624 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:45.860911 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.860887 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:45.861633 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.861614 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:51:45.892854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.892641 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:45.893152 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.893129 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:45.894572 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:45.894553 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:51:46.664910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:46.664868 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7165-17251b"] Oct 31 03:51:46.670861 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:46.670826 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7165-17251b"] Oct 31 03:51:47.062259 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:47.062164 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7165-17251b" podUID="2148d4f1-2bfb-4de1-9646-94bea2440ff6" containerName="test-curl-7165-17251b" containerID="cri-o://df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089" gracePeriod=2 Oct 31 03:51:49.166008 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:51:49.165962 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:51:49.341139 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:49.341113 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:49.455896 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:49.455864 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2148d4f1-2bfb-4de1-9646-94bea2440ff6-kube-api-access-qkfj7\" (UniqueName: \"kubernetes.io/projected/2148d4f1-2bfb-4de1-9646-94bea2440ff6-kube-api-access-qkfj7\") pod \"2148d4f1-2bfb-4de1-9646-94bea2440ff6\" (UID: \"2148d4f1-2bfb-4de1-9646-94bea2440ff6\") " Oct 31 03:51:49.458109 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:49.458075 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2148d4f1-2bfb-4de1-9646-94bea2440ff6-kube-api-access-qkfj7" pod "2148d4f1-2bfb-4de1-9646-94bea2440ff6" (UID: "2148d4f1-2bfb-4de1-9646-94bea2440ff6"). InnerVolumeSpecName "kube-api-access-qkfj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:49.557673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:49.557636 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkfj7\" (UniqueName: \"kubernetes.io/projected/2148d4f1-2bfb-4de1-9646-94bea2440ff6-kube-api-access-qkfj7\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:50.094306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:50.094269 4934 generic.go:356] "Generic (PLEG): container finished" podID="2148d4f1-2bfb-4de1-9646-94bea2440ff6" containerID="df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089" exitCode=137 Oct 31 03:51:50.094489 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:50.094321 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:50.094489 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:50.094376 4934 scope.go:122] "RemoveContainer" containerID="df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089" Oct 31 03:51:50.106992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:50.106913 4934 scope.go:122] "RemoveContainer" containerID="df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089" Oct 31 03:51:50.107348 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:51:50.107312 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089\": container with ID starting with df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089 not found: ID does not exist" containerID="df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089" Oct 31 03:51:50.107437 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:50.107361 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089"} err="failed to get container status \"df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089\": rpc error: code = NotFound desc = could not find container \"df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089\": container with ID starting with df75595a3ce84ebdf64b094678510bf112cc3e1436a71056a22d7f36855dc089 not found: ID does not exist" Oct 31 03:51:50.158636 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:51:50.158597 4934 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7165-17251b\" is forbidden: User \"system:node:ip-10-0-138-238.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-238.us-west-1.compute.internal' and this object" podUID="2148d4f1-2bfb-4de1-9646-94bea2440ff6" pod="opendatahub/test-curl-7165-17251b" Oct 31 03:51:50.821512 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:50.821483 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2148d4f1-2bfb-4de1-9646-94bea2440ff6" path="/var/lib/kubelet/pods/2148d4f1-2bfb-4de1-9646-94bea2440ff6/volumes" Oct 31 03:51:59.674080 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.674042 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8"] Oct 31 03:51:59.674717 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.674700 4934 cpu_manager.go:414] "containerMap: removing container" podUID="2148d4f1-2bfb-4de1-9646-94bea2440ff6" containerName="test-curl-7165-17251b" Oct 31 03:51:59.674785 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.674718 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2148d4f1-2bfb-4de1-9646-94bea2440ff6" containerName="test-curl-7165-17251b" Oct 31 03:51:59.674848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.674819 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="2148d4f1-2bfb-4de1-9646-94bea2440ff6" containerName="test-curl-7165-17251b" Oct 31 03:51:59.679592 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.679564 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.699840 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-socket\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.699949 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istiod-ca-cert\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.700003 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kdz\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-kube-api-access-66kdz\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.700049 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-podinfo\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.700078 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-credential-socket\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.700104 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-data\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.700244 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-envoy\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.700276 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-token\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.701288 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.700342 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-certs\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.737636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.737599 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-05a9ff-openshift-default-dockercfg-l7rk7\"" Oct 31 03:51:59.872811 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.872754 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-envoy\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.874054 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.874030 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-token\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.875015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.874970 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-certs\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.875158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.874664 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-certs\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.905414 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.905382 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-socket\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.905854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.905836 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istiod-ca-cert\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.906031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.906005 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kdz\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-kube-api-access-66kdz\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.906205 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.906192 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-podinfo\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.906344 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.906330 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-credential-socket\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.906510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.906482 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-data\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.908043 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.908023 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-data\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.909004 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.908659 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istiod-ca-cert\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.912466 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.912416 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-podinfo\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.922754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.922682 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-credential-socket\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.922754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.922689 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-socket\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.926892 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.926756 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8"] Oct 31 03:51:59.983711 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.983619 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-envoy\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.990671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.990514 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-token\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:51:59.990858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:51:59.990805 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kdz\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-kube-api-access-66kdz\") pod \"e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:52:00.003356 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:00.003323 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:52:00.616987 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:00.616955 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8"] Oct 31 03:52:00.618003 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:52:00.617978 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365b8a0d_2476_4e6f_95fa_ddcbad399dca.slice/crio-73b4f18938454dc0deafa7f77167f9474256fa4689a2cc644339ec46aa796d91 WatchSource:0}: Error finding container 73b4f18938454dc0deafa7f77167f9474256fa4689a2cc644339ec46aa796d91: Status 404 returned error can't find the container with id 73b4f18938454dc0deafa7f77167f9474256fa4689a2cc644339ec46aa796d91 Oct 31 03:52:00.620866 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:00.620832 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:00.620944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:00.620901 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:00.620944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:00.620929 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:01.279914 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:01.279881 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" event={"ID":"365b8a0d-2476-4e6f-95fa-ddcbad399dca","Type":"ContainerStarted","Data":"58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68"} Oct 31 03:52:01.280365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:01.279933 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" event={"ID":"365b8a0d-2476-4e6f-95fa-ddcbad399dca","Type":"ContainerStarted","Data":"73b4f18938454dc0deafa7f77167f9474256fa4689a2cc644339ec46aa796d91"} Oct 31 03:52:01.403949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:01.403899 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podStartSLOduration=2.4038849239999998 podStartE2EDuration="2.403884924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:01.401031744 +0000 UTC m=+1954.719774371" watchObservedRunningTime="2026-10-31 03:52:01.403884924 +0000 UTC m=+1954.722627551" Oct 31 03:52:02.004804 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:02.004771 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:52:02.010351 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:02.010324 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:52:02.286540 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:02.286435 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:52:02.288003 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:02.287981 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:52:19.164064 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:52:19.164029 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:52:19.416627 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:19.416551 4934 generic.go:356] "Generic (PLEG): container finished" podID="25a196b3-a77a-414a-9995-b766598757aa" containerID="0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3" exitCode=0 Oct 31 03:52:19.416749 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:19.416620 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" event={"ID":"25a196b3-a77a-414a-9995-b766598757aa","Type":"ContainerDied","Data":"0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3"} Oct 31 03:52:19.417031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:19.417016 4934 scope.go:122] "RemoveContainer" containerID="0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3" Oct 31 03:52:19.418876 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:19.418844 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:19.418973 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:19.418934 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:19.419013 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:19.418979 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:20.424498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:20.424467 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" event={"ID":"25a196b3-a77a-414a-9995-b766598757aa","Type":"ContainerStarted","Data":"95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f"} Oct 31 03:52:20.504865 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:20.504836 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:52:20.505103 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:20.505081 4934 patch_prober.go:29] interesting pod/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:52:20.505155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:20.505136 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:52:21.505178 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:21.505146 4934 patch_prober.go:29] interesting pod/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:52:21.505593 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:21.505215 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:52:22.504813 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:22.504760 4934 patch_prober.go:29] interesting pod/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:52:22.505023 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:22.504834 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:52:23.505052 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.505009 4934 patch_prober.go:29] interesting pod/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:52:23.505556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.505072 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:52:23.848479 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.848447 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl"] Oct 31 03:52:23.852682 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.852663 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.990594 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990560 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2298e006-07b2-4516-84f3-1151da2a2991-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.990805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990659 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2298e006-07b2-4516-84f3-1151da2a2991-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.990805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990682 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.990805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990722 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.990805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990774 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.990805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990791 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.990805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990810 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.991152 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990829 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cpb\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-kube-api-access-p9cpb\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:23.991152 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:23.990856 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.095697 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.095658 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl"] Oct 31 03:52:24.096825 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.096792 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2298e006-07b2-4516-84f3-1151da2a2991-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.098820 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.098760 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2298e006-07b2-4516-84f3-1151da2a2991-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.098991 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.098970 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.099151 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.099133 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.099365 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.099347 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.099484 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.099469 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.099597 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.099582 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.099705 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.099689 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cpb\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-kube-api-access-p9cpb\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.099825 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.099811 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.101019 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.100997 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2298e006-07b2-4516-84f3-1151da2a2991-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.102922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.101909 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.103061 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.102123 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.103169 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.102381 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.103304 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.102869 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.117311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.117212 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.119939 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.119912 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2298e006-07b2-4516-84f3-1151da2a2991-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.186776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.186741 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.187008 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.186857 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cpb\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-kube-api-access-p9cpb\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.483317 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.480477 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:24.505593 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.505566 4934 patch_prober.go:29] interesting pod/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:52:24.505902 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.505628 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:52:24.916155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.916125 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl"] Oct 31 03:52:24.917739 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:52:24.917712 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2298e006_07b2_4516_84f3_1151da2a2991.slice/crio-602e54d217d334c66f16809e974a70857f027e32b9627c1fbcedc3d7f5da39d7 WatchSource:0}: Error finding container 602e54d217d334c66f16809e974a70857f027e32b9627c1fbcedc3d7f5da39d7: Status 404 returned error can't find the container with id 602e54d217d334c66f16809e974a70857f027e32b9627c1fbcedc3d7f5da39d7 Oct 31 03:52:24.920400 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.920374 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:24.920452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.920434 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:24.920488 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:24.920460 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:25.456789 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:25.456752 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" event={"ID":"2298e006-07b2-4516-84f3-1151da2a2991","Type":"ContainerStarted","Data":"6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd"} Oct 31 03:52:25.457080 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:25.457052 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" event={"ID":"2298e006-07b2-4516-84f3-1151da2a2991","Type":"ContainerStarted","Data":"602e54d217d334c66f16809e974a70857f027e32b9627c1fbcedc3d7f5da39d7"} Oct 31 03:52:25.480949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:25.480924 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:25.509147 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:25.509122 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:52:25.509598 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:25.509467 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:52:25.510016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:25.509997 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:52:25.679725 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:25.679668 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" podStartSLOduration=2.679652821 podStartE2EDuration="2.679652821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:25.627716199 +0000 UTC m=+1978.946458826" watchObservedRunningTime="2026-10-31 03:52:25.679652821 +0000 UTC m=+1978.998395447" Oct 31 03:52:26.495311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:26.490562 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:52:26.495311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:26.490625 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 03:52:27.485358 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:27.485323 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:52:27.485895 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:27.485866 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 03:52:28.483555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:28.483519 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:52:28.484372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:28.484326 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 03:52:29.481548 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:29.481514 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:52:29.482166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:29.482118 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 03:52:30.482957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:30.482919 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:52:30.483586 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:30.483553 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 03:52:31.488024 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:31.487987 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:31.488747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:31.488728 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:31.489685 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:31.489661 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:52:45.166944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:45.166907 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6"] Oct 31 03:52:45.167824 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:45.167791 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" containerID="cri-o://e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6" gracePeriod=30 Oct 31 03:52:49.168221 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:52:49.168173 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:52:50.078129 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.078089 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv"] Oct 31 03:52:50.082319 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.082294 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.136639 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.136611 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e02dd5-serving-cert\"" Oct 31 03:52:50.163794 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.163758 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.163982 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.163954 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-metrics-certs\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.164058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.164008 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4d9\" (UniqueName: \"kubernetes.io/projected/840e5980-4118-4f8d-988b-2920003c251c-kube-api-access-pj4d9\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.164153 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.164136 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840e5980-4118-4f8d-988b-2920003c251c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.189617 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.189592 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv"] Oct 31 03:52:50.265829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.265798 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.265995 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:52:50.265944 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-serving-cert: secret "maas-api-e2e-api-b-e02dd5-serving-cert" not found Oct 31 03:52:50.265995 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.265956 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-metrics-certs\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.266097 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:52:50.266007 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls podName:840e5980-4118-4f8d-988b-2920003c251c nodeName:}" failed. No retries permitted until 2026-10-31 03:52:50.765987273 +0000 UTC m=+2004.084729904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls") pod "maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" (UID: "840e5980-4118-4f8d-988b-2920003c251c") : secret "maas-api-e2e-api-b-e02dd5-serving-cert" not found Oct 31 03:52:50.266097 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.266025 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4d9\" (UniqueName: \"kubernetes.io/projected/840e5980-4118-4f8d-988b-2920003c251c-kube-api-access-pj4d9\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.266261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.266155 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840e5980-4118-4f8d-988b-2920003c251c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.266842 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.266817 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840e5980-4118-4f8d-988b-2920003c251c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.268409 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.268389 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-metrics-certs\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.327875 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.327831 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4d9\" (UniqueName: \"kubernetes.io/projected/840e5980-4118-4f8d-988b-2920003c251c-kube-api-access-pj4d9\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.442403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.442381 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:52:50.571706 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.571666 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-podinfo\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.571912 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.571745 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-socket\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.571912 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.571778 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-kube-api-access-jvnqd\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-kube-api-access-jvnqd\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.572033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.571998 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-token\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-token\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.572113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.572077 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-socket" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:50.572174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.572032 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-certs\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.572174 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.572154 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-data\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.572281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.572201 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-envoy\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.572366 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.572343 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6878af03-5f4c-467c-bbb5-01fe97ade244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6878af03-5f4c-467c-bbb5-01fe97ade244-istiod-ca-cert\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.572449 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.572434 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-credential-socket\") pod \"6878af03-5f4c-467c-bbb5-01fe97ade244\" (UID: \"6878af03-5f4c-467c-bbb5-01fe97ade244\") " Oct 31 03:52:50.573422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.573145 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.573422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.573273 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-certs" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:50.573422 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.573169 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6878af03-5f4c-467c-bbb5-01fe97ade244-istiod-ca-cert" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:50.573665 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.573456 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-data" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:50.573806 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.573772 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-credential-socket" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:50.575419 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.575394 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-kube-api-access-jvnqd" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "kube-api-access-jvnqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:50.575521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.575432 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-token" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:50.576244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.576209 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-podinfo" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:52:50.576554 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.576536 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-envoy" pod "6878af03-5f4c-467c-bbb5-01fe97ade244" (UID: "6878af03-5f4c-467c-bbb5-01fe97ade244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:50.675065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.674983 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6878af03-5f4c-467c-bbb5-01fe97ade244-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.675244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.675107 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.675244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.675128 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.675244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.675141 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnqd\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-kube-api-access-jvnqd\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.675244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.675157 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.675244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.675192 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.675244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.675206 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.675244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.675220 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6878af03-5f4c-467c-bbb5-01fe97ade244-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:50.688947 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.688919 4934 generic.go:356] "Generic (PLEG): container finished" podID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerID="e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6" exitCode=0 Oct 31 03:52:50.689094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.688975 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" Oct 31 03:52:50.689094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.689006 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" event={"ID":"6878af03-5f4c-467c-bbb5-01fe97ade244","Type":"ContainerDied","Data":"e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6"} Oct 31 03:52:50.689094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.689062 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6" event={"ID":"6878af03-5f4c-467c-bbb5-01fe97ade244","Type":"ContainerDied","Data":"3b1fb6c8d9bd46a0d9c0869ba9512ae0c8d98b026d59c96194f192a3e781033f"} Oct 31 03:52:50.689094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.689087 4934 scope.go:122] "RemoveContainer" containerID="e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6" Oct 31 03:52:50.699468 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.699445 4934 scope.go:122] "RemoveContainer" containerID="e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6" Oct 31 03:52:50.700037 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:52:50.699930 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6\": container with ID starting with e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6 not found: ID does not exist" containerID="e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6" Oct 31 03:52:50.700037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.699968 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6"} err="failed to get container status \"e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6\": rpc error: code = NotFound desc = could not find container \"e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6\": container with ID starting with e4999feea0f01830df26cdb53366fd3817e9f82fc360582b4c655af56a73fcd6 not found: ID does not exist" Oct 31 03:52:50.776258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.776187 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.778698 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.778675 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls\") pod \"maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:50.855720 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.855691 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6"] Oct 31 03:52:50.861206 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.861179 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-djvn6"] Oct 31 03:52:50.997522 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:50.997440 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:51.383921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:51.383879 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv"] Oct 31 03:52:51.385778 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:52:51.385740 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840e5980_4118_4f8d_988b_2920003c251c.slice/crio-bc72770763e49ce2ae557bece25cdbf157c41799272c87002da4265914958a57 WatchSource:0}: Error finding container bc72770763e49ce2ae557bece25cdbf157c41799272c87002da4265914958a57: Status 404 returned error can't find the container with id bc72770763e49ce2ae557bece25cdbf157c41799272c87002da4265914958a57 Oct 31 03:52:51.698580 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:51.698475 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" event={"ID":"840e5980-4118-4f8d-988b-2920003c251c","Type":"ContainerStarted","Data":"bc72770763e49ce2ae557bece25cdbf157c41799272c87002da4265914958a57"} Oct 31 03:52:52.708555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:52.708465 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" event={"ID":"840e5980-4118-4f8d-988b-2920003c251c","Type":"ContainerStarted","Data":"2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6"} Oct 31 03:52:52.708899 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:52.708597 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:52:52.822010 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:52.821976 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" path="/var/lib/kubelet/pods/6878af03-5f4c-467c-bbb5-01fe97ade244/volumes" Oct 31 03:52:52.829875 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:52.829832 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" podStartSLOduration=1.8489378159999998 podStartE2EDuration="2.829818411s" totalImagesPullingTime="980.880595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:52.827072008 +0000 UTC m=+2006.145814635" watchObservedRunningTime="2026-10-31 03:52:52.829818411 +0000 UTC m=+2006.148561046" Oct 31 03:52:54.175091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.175057 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm"] Oct 31 03:52:54.175698 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.175677 4934 cpu_manager.go:414] "containerMap: removing container" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" Oct 31 03:52:54.175698 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.175697 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" Oct 31 03:52:54.175912 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.175876 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="6878af03-5f4c-467c-bbb5-01fe97ade244" containerName="istio-proxy" Oct 31 03:52:54.179552 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.179533 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.236264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.234291 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b16de9-openshift-default-dockercfg-g4k4r\"" Oct 31 03:52:54.298262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.297447 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm"] Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355141 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-certs\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355254 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-credential-socket\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355302 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c01246-a320-4e53-97f9-f26a2591547c-istiod-ca-cert\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355375 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dr5k\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-kube-api-access-6dr5k\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355443 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c01246-a320-4e53-97f9-f26a2591547c-istio-podinfo\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355495 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-data\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355531 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-socket\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355553 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-envoy\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.355586 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-istio-token\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.477993 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-certs\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478102 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-credential-socket\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478175 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c01246-a320-4e53-97f9-f26a2591547c-istiod-ca-cert\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478264 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dr5k\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-kube-api-access-6dr5k\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478517 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478341 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c01246-a320-4e53-97f9-f26a2591547c-istio-podinfo\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478517 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478403 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-data\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478517 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478446 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-socket\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478517 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478475 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-envoy\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.478517 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.478513 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-istio-token\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.479933 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.479901 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-socket\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.481209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.480145 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-data\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.481209 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.481011 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c01246-a320-4e53-97f9-f26a2591547c-istiod-ca-cert\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.481405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.481258 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-credential-socket\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.484438 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.482344 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-certs\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.512891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.512785 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c01246-a320-4e53-97f9-f26a2591547c-istio-podinfo\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.515281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.514598 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-envoy\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.539917 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.539883 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-istio-token\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.542499 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.542476 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dr5k\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-kube-api-access-6dr5k\") pod \"e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:54.795751 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:54.795658 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:55.249220 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:52:55.249189 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c01246_a320_4e53_97f9_f26a2591547c.slice/crio-71dcd9f3878e98cc3eeece2c05f4168e3b571dbdb8aa39feabc3a49a38fde6ba WatchSource:0}: Error finding container 71dcd9f3878e98cc3eeece2c05f4168e3b571dbdb8aa39feabc3a49a38fde6ba: Status 404 returned error can't find the container with id 71dcd9f3878e98cc3eeece2c05f4168e3b571dbdb8aa39feabc3a49a38fde6ba Oct 31 03:52:55.249776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.249691 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm"] Oct 31 03:52:55.251944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.251900 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:55.252058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.251981 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:55.252058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.252024 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:52:55.735542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.735500 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" event={"ID":"37c01246-a320-4e53-97f9-f26a2591547c","Type":"ContainerStarted","Data":"37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a"} Oct 31 03:52:55.735745 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.735565 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" event={"ID":"37c01246-a320-4e53-97f9-f26a2591547c","Type":"ContainerStarted","Data":"71dcd9f3878e98cc3eeece2c05f4168e3b571dbdb8aa39feabc3a49a38fde6ba"} Oct 31 03:52:55.796088 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.796058 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:55.858592 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:55.858544 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podStartSLOduration=1.858529491 podStartE2EDuration="1.858529491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:55.856333677 +0000 UTC m=+2009.175076313" watchObservedRunningTime="2026-10-31 03:52:55.858529491 +0000 UTC m=+2009.177272119" Oct 31 03:52:56.801844 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:56.801819 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:57.747981 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:57.747946 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:57.749117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:57.749096 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:52:58.730250 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:52:58.729283 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:53:02.310800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:02.310761 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7"] Oct 31 03:53:02.311450 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:02.311002 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" containerID="cri-o://21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290" gracePeriod=30 Oct 31 03:53:02.651490 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:02.651404 4934 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:53:02.651746 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:02.651718 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:53:07.579921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.579856 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:53:07.749200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749171 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916dada2-c7fc-4667-8682-fabab974e046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916dada2-c7fc-4667-8682-fabab974e046-istiod-ca-cert\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749270 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/916dada2-c7fc-4667-8682-fabab974e046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916dada2-c7fc-4667-8682-fabab974e046-istio-podinfo\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749377 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-data\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749429 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-certs\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749553 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749469 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-kube-api-access-z6dzq\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-kube-api-access-z6dzq\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749607 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749572 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-istio-token\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-istio-token\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749663 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749626 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-envoy\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749736 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749713 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-credential-socket\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.749823 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749805 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-socket\") pod \"916dada2-c7fc-4667-8682-fabab974e046\" (UID: \"916dada2-c7fc-4667-8682-fabab974e046\") " Oct 31 03:53:07.750044 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749714 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916dada2-c7fc-4667-8682-fabab974e046-istiod-ca-cert" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:07.750124 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.749847 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-data" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:07.750124 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750046 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-certs" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:07.750124 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750098 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-credential-socket" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:07.750419 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750390 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-socket" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:07.750900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750875 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.751001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750905 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916dada2-c7fc-4667-8682-fabab974e046-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.751001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750921 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.751001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750935 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.751001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.750949 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.752127 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.751990 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/916dada2-c7fc-4667-8682-fabab974e046-istio-podinfo" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:07.753085 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.753062 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-envoy" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:07.753378 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.753353 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-istio-token" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:07.753458 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.753411 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-kube-api-access-z6dzq" pod "916dada2-c7fc-4667-8682-fabab974e046" (UID: "916dada2-c7fc-4667-8682-fabab974e046"). InnerVolumeSpecName "kube-api-access-z6dzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:07.834630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.834558 4934 generic.go:356] "Generic (PLEG): container finished" podID="916dada2-c7fc-4667-8682-fabab974e046" containerID="21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290" exitCode=0 Oct 31 03:53:07.834630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.834603 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" Oct 31 03:53:07.834776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.834641 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" event={"ID":"916dada2-c7fc-4667-8682-fabab974e046","Type":"ContainerDied","Data":"21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290"} Oct 31 03:53:07.834776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.834693 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7" event={"ID":"916dada2-c7fc-4667-8682-fabab974e046","Type":"ContainerDied","Data":"b4921d6a67812ca78d18ffd5b7d2cb8164f77d8333e4bd356cbba812346bd92f"} Oct 31 03:53:07.834776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.834716 4934 scope.go:122] "RemoveContainer" containerID="21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290" Oct 31 03:53:07.848395 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.848373 4934 scope.go:122] "RemoveContainer" containerID="21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290" Oct 31 03:53:07.848821 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:53:07.848802 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290\": container with ID starting with 21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290 not found: ID does not exist" containerID="21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290" Oct 31 03:53:07.848893 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.848829 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290"} err="failed to get container status \"21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290\": rpc error: code = NotFound desc = could not find container \"21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290\": container with ID starting with 21e816188a155fdb244e6b2933110926fba6a342f6dab53f34f6c3493d99b290 not found: ID does not exist" Oct 31 03:53:07.852872 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.852848 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916dada2-c7fc-4667-8682-fabab974e046-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.852953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.852876 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6dzq\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-kube-api-access-z6dzq\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.852953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.852888 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916dada2-c7fc-4667-8682-fabab974e046-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:07.852953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:07.852899 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916dada2-c7fc-4667-8682-fabab974e046-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:53:08.000290 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:08.000259 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7"] Oct 31 03:53:08.005926 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:08.005897 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-zfhf7"] Oct 31 03:53:08.829813 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:08.829780 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916dada2-c7fc-4667-8682-fabab974e046" path="/var/lib/kubelet/pods/916dada2-c7fc-4667-8682-fabab974e046/volumes" Oct 31 03:53:19.170674 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:53:19.170629 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:53:21.883440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:21.883411 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6"] Oct 31 03:53:21.884083 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:21.884063 4934 cpu_manager.go:414] "containerMap: removing container" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" Oct 31 03:53:21.884126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:21.884087 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" Oct 31 03:53:21.884293 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:21.884281 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="916dada2-c7fc-4667-8682-fabab974e046" containerName="istio-proxy" Oct 31 03:53:21.887843 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:21.887827 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:21.940704 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:21.940677 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-05a9ff-serving-cert\"" Oct 31 03:53:21.999629 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:21.999593 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6"] Oct 31 03:53:22.030552 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.030525 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762ead81-eb44-4059-a19e-d6704ea7a111-openshift-service-ca\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.030770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.030748 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.030814 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.030786 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-metrics-certs\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.030873 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.030814 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbpc\" (UniqueName: \"kubernetes.io/projected/762ead81-eb44-4059-a19e-d6704ea7a111-kube-api-access-9xbpc\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.135310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.132212 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.135310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.132292 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-metrics-certs\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.135310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.132324 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbpc\" (UniqueName: \"kubernetes.io/projected/762ead81-eb44-4059-a19e-d6704ea7a111-kube-api-access-9xbpc\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.135310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.132558 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762ead81-eb44-4059-a19e-d6704ea7a111-openshift-service-ca\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.135310 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.133592 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762ead81-eb44-4059-a19e-d6704ea7a111-openshift-service-ca\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.135310 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:53:22.133724 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-serving-cert: secret "maas-api-e2e-api-b-05a9ff-serving-cert" not found Oct 31 03:53:22.135310 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:53:22.133814 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls podName:762ead81-eb44-4059-a19e-d6704ea7a111 nodeName:}" failed. No retries permitted until 2026-10-31 03:53:22.633794219 +0000 UTC m=+2035.952536824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls") pod "maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" (UID: "762ead81-eb44-4059-a19e-d6704ea7a111") : secret "maas-api-e2e-api-b-05a9ff-serving-cert" not found Oct 31 03:53:22.144958 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.144923 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-metrics-certs\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.197845 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.197817 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbpc\" (UniqueName: \"kubernetes.io/projected/762ead81-eb44-4059-a19e-d6704ea7a111-kube-api-access-9xbpc\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.638718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.638687 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.641324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.641299 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls\") pod \"maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:22.800838 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:22.800813 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:23.186101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:23.186073 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6"] Oct 31 03:53:23.187495 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:53:23.187468 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762ead81_eb44_4059_a19e_d6704ea7a111.slice/crio-666273ccce31ba59b8d71e36c4e1a1ded5c71ddb5d51b96b64490d500284c314 WatchSource:0}: Error finding container 666273ccce31ba59b8d71e36c4e1a1ded5c71ddb5d51b96b64490d500284c314: Status 404 returned error can't find the container with id 666273ccce31ba59b8d71e36c4e1a1ded5c71ddb5d51b96b64490d500284c314 Oct 31 03:53:23.932765 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:23.932734 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" event={"ID":"762ead81-eb44-4059-a19e-d6704ea7a111","Type":"ContainerStarted","Data":"666273ccce31ba59b8d71e36c4e1a1ded5c71ddb5d51b96b64490d500284c314"} Oct 31 03:53:24.948156 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:24.945931 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" event={"ID":"762ead81-eb44-4059-a19e-d6704ea7a111","Type":"ContainerStarted","Data":"256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0"} Oct 31 03:53:24.948156 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:24.947063 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:25.067490 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:25.067430 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" podStartSLOduration=3.2502086820000002 podStartE2EDuration="4.067412877s" totalImagesPullingTime="817.204195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:25.066368426 +0000 UTC m=+2038.385111054" watchObservedRunningTime="2026-10-31 03:53:25.067412877 +0000 UTC m=+2038.386155504" Oct 31 03:53:31.981868 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:31.981834 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:53:42.090256 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:42.085859 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 03:53:42.209255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:42.208289 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.439530 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:42.439432 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:42.439785 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:42.439755 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.658391 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:42.658346 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.767432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:42.767331 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:42.767693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:42.767659 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:43.052451 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.052354 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:43.086347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.086299 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:43.086953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.086930 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:53:43.245255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.240824 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:43.248582 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.248561 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:53:43.659541 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.659500 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:53:43.665828 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.665803 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:53:43.683857 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.682477 4934 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": context deadline exceeded" Oct 31 03:53:43.738247 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.730786 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:53:43.747912 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:43.747878 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:53:44.136010 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:44.135163 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:53:46.954458 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:46.954409 4934 patch_prober.go:29] interesting pod/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:46.954992 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:46.954487 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:48.329382 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:48.329344 4934 patch_prober.go:29] interesting pod/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:48.329800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:48.329409 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:49.165802 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:53:49.165767 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:53:49.268014 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:49.267934 4934 generic.go:356] "Generic (PLEG): container finished" podID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerID="3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64" exitCode=0 Oct 31 03:53:49.268176 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:49.268010 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" event={"ID":"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0","Type":"ContainerDied","Data":"3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64"} Oct 31 03:53:49.268507 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:49.268491 4934 scope.go:122] "RemoveContainer" containerID="3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64" Oct 31 03:53:49.270631 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:49.270592 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:53:49.270722 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:49.270690 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:53:49.270761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:49.270736 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:53:50.275489 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.275460 4934 generic.go:356] "Generic (PLEG): container finished" podID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerID="58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68" exitCode=0 Oct 31 03:53:50.276009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.275534 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" event={"ID":"365b8a0d-2476-4e6f-95fa-ddcbad399dca","Type":"ContainerDied","Data":"58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68"} Oct 31 03:53:50.276090 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.276014 4934 scope.go:122] "RemoveContainer" containerID="58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68" Oct 31 03:53:50.277763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.277742 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" event={"ID":"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0","Type":"ContainerStarted","Data":"cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584"} Oct 31 03:53:50.277994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.277966 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:53:50.278072 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.278056 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:53:50.278126 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.278112 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:53:50.568555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.553610 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:53:50.568555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.553802 4934 patch_prober.go:29] interesting pod/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:53:50.568555 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:50.553843 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:53:51.296562 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:51.296523 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" event={"ID":"365b8a0d-2476-4e6f-95fa-ddcbad399dca","Type":"ContainerStarted","Data":"0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7"} Oct 31 03:53:51.567316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:51.557576 4934 patch_prober.go:29] interesting pod/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:53:51.567316 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:51.557635 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:53:52.008315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:52.007737 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:53:52.008315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:52.007861 4934 patch_prober.go:29] interesting pod/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:53:52.008315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:52.007902 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 03:53:52.569430 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:52.569397 4934 patch_prober.go:29] interesting pod/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:53:52.569997 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:52.569968 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:53:53.006680 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:53.006586 4934 patch_prober.go:29] interesting pod/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:53:53.008454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:53.008423 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 03:53:53.557570 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:53.557535 4934 patch_prober.go:29] interesting pod/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:53:53.557894 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:53.557855 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:53:54.008717 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:54.008620 4934 patch_prober.go:29] interesting pod/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:53:54.009328 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:54.009297 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 03:53:54.557580 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:54.557546 4934 patch_prober.go:29] interesting pod/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:53:54.557848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:54.557821 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:53:55.007697 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:55.007603 4934 patch_prober.go:29] interesting pod/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:53:55.007948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:55.007918 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 03:53:55.558254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:55.557974 4934 patch_prober.go:29] interesting pod/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:53:55.558254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:55.558033 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:53:56.006673 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:56.006577 4934 patch_prober.go:29] interesting pod/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:53:56.006971 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:56.006937 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 03:53:56.586671 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:56.586639 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:53:56.587691 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:56.587666 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:53:56.588635 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:56.588609 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:53:57.008573 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:57.008504 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:53:57.008825 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:57.008804 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:53:57.009449 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:53:57.009432 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:54:19.167589 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:54:19.167550 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:54:27.635513 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:27.635473 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:54:27.701832 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:27.701809 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:54:30.784884 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:30.784850 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz"] Oct 31 03:54:30.785700 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:30.785664 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" containerID="cri-o://b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067" gracePeriod=30 Oct 31 03:54:31.895068 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:31.895034 4934 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:54:31.895556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:31.895111 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:54:36.062622 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.062592 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:54:36.166091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166051 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-envoy\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166083 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45eb878-d4e7-4685-866c-3c4027aa4130-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45eb878-d4e7-4685-866c-3c4027aa4130-istiod-ca-cert\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166179 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-credential-socket\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166290 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-data\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166381 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166378 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-socket\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166536 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166483 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-podinfo\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166536 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166513 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-certs\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166536 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-credential-socket" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:36.166642 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166606 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-kube-api-access-l5c2v\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-kube-api-access-l5c2v\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166744 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166659 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-token\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-token\") pod \"f45eb878-d4e7-4685-866c-3c4027aa4130\" (UID: \"f45eb878-d4e7-4685-866c-3c4027aa4130\") " Oct 31 03:54:36.166797 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166603 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45eb878-d4e7-4685-866c-3c4027aa4130-istiod-ca-cert" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:36.166797 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.166746 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-certs" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:36.167350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.167035 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-socket" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:36.167350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.167070 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-data" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:36.167350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.167259 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.167350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.167276 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45eb878-d4e7-4685-866c-3c4027aa4130-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.167350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.167303 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.167350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.167316 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.167350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.167328 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.168633 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.168609 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-envoy" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:36.169029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.169011 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-podinfo" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:36.169516 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.169497 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-kube-api-access-l5c2v" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "kube-api-access-l5c2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:36.169615 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.169534 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-token" pod "f45eb878-d4e7-4685-866c-3c4027aa4130" (UID: "f45eb878-d4e7-4685-866c-3c4027aa4130"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:36.269583 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.269411 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5c2v\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-kube-api-access-l5c2v\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.269583 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.269440 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.269583 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.269456 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.269583 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.269470 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45eb878-d4e7-4685-866c-3c4027aa4130-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.604481 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.604447 4934 generic.go:356] "Generic (PLEG): container finished" podID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerID="b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067" exitCode=0 Oct 31 03:54:36.604677 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.604513 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" Oct 31 03:54:36.604677 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.604533 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" event={"ID":"f45eb878-d4e7-4685-866c-3c4027aa4130","Type":"ContainerDied","Data":"b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067"} Oct 31 03:54:36.604677 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.604598 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz" event={"ID":"f45eb878-d4e7-4685-866c-3c4027aa4130","Type":"ContainerDied","Data":"1f9941cee97a7839cfd91f1dd64bf24776fd4d8602d3042577e980489baf0d94"} Oct 31 03:54:36.604677 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.604626 4934 scope.go:122] "RemoveContainer" containerID="b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067" Oct 31 03:54:36.614939 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.614924 4934 scope.go:122] "RemoveContainer" containerID="b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067" Oct 31 03:54:36.615208 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:54:36.615177 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067\": container with ID starting with b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067 not found: ID does not exist" containerID="b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067" Oct 31 03:54:36.615336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.615221 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067"} err="failed to get container status \"b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067\": rpc error: code = NotFound desc = could not find container \"b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067\": container with ID starting with b5e56fff0d2806722c59505cb3773229bad4e79b865989a8cbdd25fe4ae14067 not found: ID does not exist" Oct 31 03:54:36.771426 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.771390 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz"] Oct 31 03:54:36.822258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:36.822213 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-4cpwz"] Oct 31 03:54:38.822032 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:38.821998 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" path="/var/lib/kubelet/pods/f45eb878-d4e7-4685-866c-3c4027aa4130/volumes" Oct 31 03:54:49.178421 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:54:49.178378 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:54:51.767848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:51.767811 4934 generic.go:356] "Generic (PLEG): container finished" podID="37c01246-a320-4e53-97f9-f26a2591547c" containerID="37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a" exitCode=0 Oct 31 03:54:51.768296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:51.767877 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" event={"ID":"37c01246-a320-4e53-97f9-f26a2591547c","Type":"ContainerDied","Data":"37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a"} Oct 31 03:54:51.768346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:51.768314 4934 scope.go:122] "RemoveContainer" containerID="37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a" Oct 31 03:54:51.769909 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:51.769881 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:54:51.769997 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:51.769955 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:54:51.769997 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:51.769987 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:54:52.344772 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:52.344733 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn"] Oct 31 03:54:52.345015 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:52.344990 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" containerID="cri-o://4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f" gracePeriod=30 Oct 31 03:54:52.792221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:52.792130 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" event={"ID":"37c01246-a320-4e53-97f9-f26a2591547c","Type":"ContainerStarted","Data":"2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6"} Oct 31 03:54:52.796758 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:52.796721 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:54:52.796915 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:52.796856 4934 patch_prober.go:29] interesting pod/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:54:52.796915 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:52.796903 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 03:54:53.796542 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:53.796507 4934 patch_prober.go:29] interesting pod/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:54:53.796980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:53.796568 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 03:54:54.797035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:54.796996 4934 patch_prober.go:29] interesting pod/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:54:54.797548 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:54.797048 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 03:54:55.796624 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:55.796588 4934 patch_prober.go:29] interesting pod/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:54:55.796801 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:55.796649 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 03:54:56.796940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:56.796901 4934 patch_prober.go:29] interesting pod/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:54:56.797341 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:56.796963 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 03:54:57.672502 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.672469 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:54:57.720304 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720271 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68393835-77eb-4a74-99ca-4bded4593b56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68393835-77eb-4a74-99ca-4bded4593b56-istio-podinfo\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720323 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68393835-77eb-4a74-99ca-4bded4593b56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68393835-77eb-4a74-99ca-4bded4593b56-istiod-ca-cert\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720354 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-socket\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720396 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-envoy\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720423 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-credential-socket\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720472 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720459 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-certs\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720538 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-data\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720590 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-istio-token\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-istio-token\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720739 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720627 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-kube-api-access-m9gcf\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-kube-api-access-m9gcf\") pod \"68393835-77eb-4a74-99ca-4bded4593b56\" (UID: \"68393835-77eb-4a74-99ca-4bded4593b56\") " Oct 31 03:54:57.720898 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720754 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-socket" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:57.720898 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.720777 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68393835-77eb-4a74-99ca-4bded4593b56-istiod-ca-cert" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:57.721362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.721037 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-certs" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:57.721362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.721118 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-credential-socket" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:57.721362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.721291 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68393835-77eb-4a74-99ca-4bded4593b56-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.721362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.721313 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.721362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.721328 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.721362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.721341 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.721768 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.721390 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-data" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:57.723394 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.723367 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-envoy" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:57.723567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.723540 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-kube-api-access-m9gcf" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "kube-api-access-m9gcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:57.723718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.723703 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-istio-token" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:57.724374 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.724355 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68393835-77eb-4a74-99ca-4bded4593b56-istio-podinfo" pod "68393835-77eb-4a74-99ca-4bded4593b56" (UID: "68393835-77eb-4a74-99ca-4bded4593b56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:57.747899 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.747878 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:54:57.798626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.798593 4934 patch_prober.go:29] interesting pod/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:54:57.799096 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.798663 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 03:54:57.823641 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.822974 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.823641 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.823009 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68393835-77eb-4a74-99ca-4bded4593b56-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.823641 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.823023 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.823641 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.823040 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gcf\" (UniqueName: \"kubernetes.io/projected/68393835-77eb-4a74-99ca-4bded4593b56-kube-api-access-m9gcf\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.823641 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.823057 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68393835-77eb-4a74-99ca-4bded4593b56-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:57.830721 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.830682 4934 generic.go:356] "Generic (PLEG): container finished" podID="68393835-77eb-4a74-99ca-4bded4593b56" containerID="4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f" exitCode=0 Oct 31 03:54:57.830881 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.830738 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" Oct 31 03:54:57.830881 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.830766 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" event={"ID":"68393835-77eb-4a74-99ca-4bded4593b56","Type":"ContainerDied","Data":"4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f"} Oct 31 03:54:57.830881 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.830834 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn" event={"ID":"68393835-77eb-4a74-99ca-4bded4593b56","Type":"ContainerDied","Data":"6ef93d9e1b767cb3ebbc39580f9f58b8bc376fe2fdedbe3013f5da5c627c4f0f"} Oct 31 03:54:57.830881 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.830867 4934 scope.go:122] "RemoveContainer" containerID="4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f" Oct 31 03:54:57.849592 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.849514 4934 scope.go:122] "RemoveContainer" containerID="4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f" Oct 31 03:54:57.849837 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:54:57.849814 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f\": container with ID starting with 4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f not found: ID does not exist" containerID="4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f" Oct 31 03:54:57.849913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.849849 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f"} err="failed to get container status \"4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f\": rpc error: code = NotFound desc = could not find container \"4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f\": container with ID starting with 4c6425300df3f3590bf6a5d980d31445f639870c36a13d68efde3a99c2e23f6f not found: ID does not exist" Oct 31 03:54:57.960252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.960156 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn"] Oct 31 03:54:57.961891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:57.961869 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-xs9wn"] Oct 31 03:54:58.801203 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:58.801173 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:54:58.802213 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:58.802191 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:54:58.822531 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:54:58.822506 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68393835-77eb-4a74-99ca-4bded4593b56" path="/var/lib/kubelet/pods/68393835-77eb-4a74-99ca-4bded4593b56/volumes" Oct 31 03:55:19.167435 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:55:19.167391 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:55:49.164026 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:55:49.163990 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:55:54.397786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.397747 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb"] Oct 31 03:55:54.398545 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.398523 4934 cpu_manager.go:414] "containerMap: removing container" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" Oct 31 03:55:54.398654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.398546 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" Oct 31 03:55:54.398654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.398567 4934 cpu_manager.go:414] "containerMap: removing container" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" Oct 31 03:55:54.398654 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.398575 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" Oct 31 03:55:54.398803 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.398761 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45eb878-d4e7-4685-866c-3c4027aa4130" containerName="istio-proxy" Oct 31 03:55:54.398803 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.398777 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="68393835-77eb-4a74-99ca-4bded4593b56" containerName="istio-proxy" Oct 31 03:55:54.402287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.402267 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.454475 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.454447 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1180d818-openshift-default-dockercfg-gjfz4\"" Oct 31 03:55:54.492838 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.492808 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.492969 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.492945 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06443ca-871f-413c-885f-200727268117-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.493019 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.492977 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhrz\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-kube-api-access-5qhrz\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.493019 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.493007 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.493094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.493038 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.493094 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.493062 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.493161 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.493109 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.493220 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.493200 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06443ca-871f-413c-885f-200727268117-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.493440 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.493420 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595076 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595039 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595303 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595154 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06443ca-871f-413c-885f-200727268117-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595303 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595198 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595303 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595295 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595441 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06443ca-871f-413c-885f-200727268117-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595486 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhrz\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-kube-api-access-5qhrz\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595595 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595514 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595595 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595548 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595595 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595572 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595744 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595626 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.595744 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595731 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.596171 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.595877 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.596171 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.596130 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.596515 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.596495 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06443ca-871f-413c-885f-200727268117-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.597513 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.597486 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.597757 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.597737 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06443ca-871f-413c-885f-200727268117-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.605338 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.605314 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb"] Oct 31 03:55:54.657667 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.657598 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.657840 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.657676 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhrz\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-kube-api-access-5qhrz\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:54.715695 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:54.715667 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:55.149364 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.149342 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb"] Oct 31 03:55:55.150479 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:55:55.150457 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06443ca_871f_413c_885f_200727268117.slice/crio-386105d8bd98f7da37f292a377858e8cba6a5ef6c6a10ea7b2db36cabe5a3c69 WatchSource:0}: Error finding container 386105d8bd98f7da37f292a377858e8cba6a5ef6c6a10ea7b2db36cabe5a3c69: Status 404 returned error can't find the container with id 386105d8bd98f7da37f292a377858e8cba6a5ef6c6a10ea7b2db36cabe5a3c69 Oct 31 03:55:55.152879 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.152848 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:55:55.152949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.152913 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:55:55.152949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.152942 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:55:55.273478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.273433 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" event={"ID":"a06443ca-871f-413c-885f-200727268117","Type":"ContainerStarted","Data":"0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8"} Oct 31 03:55:55.273647 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.273497 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" event={"ID":"a06443ca-871f-413c-885f-200727268117","Type":"ContainerStarted","Data":"386105d8bd98f7da37f292a377858e8cba6a5ef6c6a10ea7b2db36cabe5a3c69"} Oct 31 03:55:55.397552 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.397460 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" podStartSLOduration=1.397441697 podStartE2EDuration="1.397441697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:55.394517134 +0000 UTC m=+2188.713259763" watchObservedRunningTime="2026-10-31 03:55:55.397441697 +0000 UTC m=+2188.716184324" Oct 31 03:55:55.716800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:55.716716 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:55:56.717771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:56.717742 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:55:56.718192 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:56.717791 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 03:55:57.716465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:57.716426 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:55:57.716655 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:57.716484 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 03:55:58.716007 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:58.715961 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:55:58.716459 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:58.716038 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 03:55:59.716520 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:59.716480 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:55:59.716965 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:55:59.716551 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 03:56:00.715976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:00.715943 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:56:00.716165 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:00.716014 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 03:56:01.720330 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:01.720220 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:56:01.720720 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:01.720632 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:56:01.721373 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:01.721358 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:56:09.521452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.521377 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg"] Oct 31 03:56:09.525941 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.525920 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.627622 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.627584 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.627826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.627651 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9cc\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-kube-api-access-2c9cc\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.627826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.627794 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.627953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.627860 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.627953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.627885 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.628050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.627976 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.628107 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.628058 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.628107 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.628090 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.628244 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.628206 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.633058 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.633033 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg"] Oct 31 03:56:09.729624 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729595 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.729829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729654 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.729829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729716 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.729829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729754 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9cc\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-kube-api-access-2c9cc\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.729829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729779 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.729829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729805 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.729829 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729822 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.730141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729889 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.730141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.729942 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.730141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.730088 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.730400 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.730174 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.730400 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.730332 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.730400 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.730379 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.730586 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.730566 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.732276 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.732250 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.732718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.732697 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.791556 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.791441 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.791841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.791817 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9cc\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-kube-api-access-2c9cc\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:09.868726 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:09.868686 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:10.304865 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.304829 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg"] Oct 31 03:56:10.306294 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:56:10.306217 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b7182b_43bf_4208_8836_ce6eb34a5a14.slice/crio-d1fc097e7bd058acee0474c6bb845384620d8e2ba72175b360c5ca6431cf6912 WatchSource:0}: Error finding container d1fc097e7bd058acee0474c6bb845384620d8e2ba72175b360c5ca6431cf6912: Status 404 returned error can't find the container with id d1fc097e7bd058acee0474c6bb845384620d8e2ba72175b360c5ca6431cf6912 Oct 31 03:56:10.309196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.309164 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:56:10.309280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.309247 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:56:10.309320 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.309278 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:56:10.384090 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.384057 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" event={"ID":"f6b7182b-43bf-4208-8836-ce6eb34a5a14","Type":"ContainerStarted","Data":"b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138"} Oct 31 03:56:10.384280 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.384121 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" event={"ID":"f6b7182b-43bf-4208-8836-ce6eb34a5a14","Type":"ContainerStarted","Data":"d1fc097e7bd058acee0474c6bb845384620d8e2ba72175b360c5ca6431cf6912"} Oct 31 03:56:10.506752 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.506701 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" podStartSLOduration=1.506685625 podStartE2EDuration="1.506685625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:10.50378217 +0000 UTC m=+2203.822524788" watchObservedRunningTime="2026-10-31 03:56:10.506685625 +0000 UTC m=+2203.825428253" Oct 31 03:56:10.869811 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:10.869776 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:11.885287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:11.879350 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:56:11.885287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:11.879413 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 03:56:12.876254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:12.871572 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:56:12.876254 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:12.871632 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 03:56:13.875255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:13.873872 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:56:13.875255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:13.873948 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 03:56:14.878720 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:14.878680 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:56:14.879143 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:14.878749 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 03:56:15.876403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:15.876342 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:56:15.876835 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:15.876810 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 03:56:16.873649 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:16.873620 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:16.874166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:16.874009 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:16.874767 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:16.874746 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:56:19.163715 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:56:19.163679 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:56:24.633619 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.633568 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb"] Oct 31 03:56:24.637632 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.637609 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.677805 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.677776 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.677920 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.677845 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.677920 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.677910 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.678033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.677936 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.678033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.678003 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.678141 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.678060 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvpn\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-kube-api-access-4gvpn\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.678212 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.678142 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.678311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.678247 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.678370 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.678306 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.744951 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.744919 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb"] Oct 31 03:56:24.779009 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.778985 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvpn\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-kube-api-access-4gvpn\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779028 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779103 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779338 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779185 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779338 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779253 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779442 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779350 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779442 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779384 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779442 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779415 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779602 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779457 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779682 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779659 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779688 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779838 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779818 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.779964 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.779941 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.780465 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.780413 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.782100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.782075 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.782372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.782351 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.840315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.840293 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.840430 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.840373 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvpn\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-kube-api-access-4gvpn\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:24.952528 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:24.952466 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:25.392375 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.392350 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb"] Oct 31 03:56:25.393175 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:56:25.393147 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8088fc_01d4_4ce9_85f4_32fd06caf3f8.slice/crio-a5e30b422a918db01ba7f6c75c05450c73361a5594cc546fc71dba259b88a165 WatchSource:0}: Error finding container a5e30b422a918db01ba7f6c75c05450c73361a5594cc546fc71dba259b88a165: Status 404 returned error can't find the container with id a5e30b422a918db01ba7f6c75c05450c73361a5594cc546fc71dba259b88a165 Oct 31 03:56:25.395786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.395754 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:56:25.395854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.395818 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:56:25.395854 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.395848 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 03:56:25.478940 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.478902 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" event={"ID":"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8","Type":"ContainerStarted","Data":"d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce"} Oct 31 03:56:25.479084 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.479002 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" event={"ID":"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8","Type":"ContainerStarted","Data":"a5e30b422a918db01ba7f6c75c05450c73361a5594cc546fc71dba259b88a165"} Oct 31 03:56:25.602565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.602512 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podStartSLOduration=1.602498276 podStartE2EDuration="1.602498276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:25.600316017 +0000 UTC m=+2218.919058670" watchObservedRunningTime="2026-10-31 03:56:25.602498276 +0000 UTC m=+2218.921240904" Oct 31 03:56:25.952815 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:25.952785 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:26.981253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:26.977350 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:56:26.981253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:26.977589 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 03:56:27.952919 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:27.952889 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:56:27.953100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:27.952943 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 03:56:28.963255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:28.957304 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:56:28.963255 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:28.957357 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 03:56:29.958627 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:29.958590 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:56:29.958921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:29.958894 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 03:56:30.961400 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:30.957523 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:56:30.961400 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:30.957583 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 03:56:32.071414 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:32.071379 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:32.072650 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:32.072612 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:32.073347 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:32.073328 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:56:40.783987 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:40.783949 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh"] Oct 31 03:56:40.784510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:40.784142 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" containerID="cri-o://ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6" gracePeriod=30 Oct 31 03:56:40.834306 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:40.834274 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl"] Oct 31 03:56:40.834528 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:40.834503 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" containerID="cri-o://6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd" gracePeriod=30 Oct 31 03:56:45.894240 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:45.894195 4934 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:56:45.894616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:45.894300 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:56:46.134027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.133986 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:56:46.165884 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.165862 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:56:46.178315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178293 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-podinfo\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178336 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-socket\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178416 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178412 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-certs\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178489 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178461 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-data\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178571 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178560 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a111075-2227-42a4-a864-54fe1fbc54a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a111075-2227-42a4-a864-54fe1fbc54a6-istiod-ca-cert\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178649 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178624 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-socket" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.178649 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178641 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-envoy\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178675 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-token\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-token\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178721 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-credential-socket\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178761 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178750 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-kube-api-access-npg8q\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-kube-api-access-npg8q\") pod \"2a111075-2227-42a4-a864-54fe1fbc54a6\" (UID: \"2a111075-2227-42a4-a864-54fe1fbc54a6\") " Oct 31 03:56:46.178985 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178728 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-certs" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.178985 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178786 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-data" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.178985 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.178927 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-credential-socket" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.179322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.179285 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a111075-2227-42a4-a864-54fe1fbc54a6-istiod-ca-cert" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:46.179578 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.179376 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a111075-2227-42a4-a864-54fe1fbc54a6-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.179578 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.179397 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.179578 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.179410 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.179578 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.179424 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.179578 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.179437 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.180951 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.180931 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-token" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:46.181217 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.181194 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-podinfo" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:46.181333 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.181222 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-kube-api-access-npg8q" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "kube-api-access-npg8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:46.181403 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.181350 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-envoy" pod "2a111075-2227-42a4-a864-54fe1fbc54a6" (UID: "2a111075-2227-42a4-a864-54fe1fbc54a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.280200 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280170 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-socket\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280366 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280273 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-data\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280366 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280298 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-envoy\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280366 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280324 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-kube-api-access-p9cpb\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-kube-api-access-p9cpb\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280569 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280412 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2298e006-07b2-4516-84f3-1151da2a2991-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2298e006-07b2-4516-84f3-1151da2a2991-istio-podinfo\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280569 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280429 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-socket" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.280569 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280466 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-istio-token\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-istio-token\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280569 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280492 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2298e006-07b2-4516-84f3-1151da2a2991-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2298e006-07b2-4516-84f3-1151da2a2991-istiod-ca-cert\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280778 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280573 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-credential-socket\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280778 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280652 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-certs\") pod \"2298e006-07b2-4516-84f3-1151da2a2991\" (UID: \"2298e006-07b2-4516-84f3-1151da2a2991\") " Oct 31 03:56:46.280778 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280658 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-data" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.280943 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.280860 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-credential-socket" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.281048 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281029 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-certs" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.281290 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281095 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2298e006-07b2-4516-84f3-1151da2a2991-istiod-ca-cert" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:46.281492 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281468 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281496 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281511 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281526 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npg8q\" (UniqueName: \"kubernetes.io/projected/2a111075-2227-42a4-a864-54fe1fbc54a6-kube-api-access-npg8q\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281540 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a111075-2227-42a4-a864-54fe1fbc54a6-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281553 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2298e006-07b2-4516-84f3-1151da2a2991-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281567 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281581 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.281931 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.281594 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.282747 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.282725 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-envoy" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:46.282888 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.282872 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-kube-api-access-p9cpb" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "kube-api-access-p9cpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:46.283535 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.283511 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-istio-token" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:46.283658 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.283541 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2298e006-07b2-4516-84f3-1151da2a2991-istio-podinfo" pod "2298e006-07b2-4516-84f3-1151da2a2991" (UID: "2298e006-07b2-4516-84f3-1151da2a2991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:46.383478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.383386 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2298e006-07b2-4516-84f3-1151da2a2991-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.383478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.383413 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9cpb\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-kube-api-access-p9cpb\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.383478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.383424 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2298e006-07b2-4516-84f3-1151da2a2991-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.383478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.383434 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2298e006-07b2-4516-84f3-1151da2a2991-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:46.565172 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.565136 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7165-ea87cd"] Oct 31 03:56:46.565792 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.565774 4934 cpu_manager.go:414] "containerMap: removing container" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" Oct 31 03:56:46.565792 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.565793 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" Oct 31 03:56:46.565963 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.565821 4934 cpu_manager.go:414] "containerMap: removing container" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" Oct 31 03:56:46.565963 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.565827 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" Oct 31 03:56:46.565963 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.565936 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="2298e006-07b2-4516-84f3-1151da2a2991" containerName="istio-proxy" Oct 31 03:56:46.565963 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.565947 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerName="istio-proxy" Oct 31 03:56:46.570623 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.570604 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:46.623476 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.623447 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7165-ea87cd"] Oct 31 03:56:46.623596 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.623462 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hppm7\"" Oct 31 03:56:46.686858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.686789 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/a0c01170-981e-423d-8529-5ae0076a99de-kube-api-access-rfnnv\") pod \"test-curl-7165-ea87cd\" (UID: \"a0c01170-981e-423d-8529-5ae0076a99de\") " pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:46.711660 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.711634 4934 generic.go:356] "Generic (PLEG): container finished" podID="2298e006-07b2-4516-84f3-1151da2a2991" containerID="6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd" exitCode=0 Oct 31 03:56:46.711784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.711675 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" Oct 31 03:56:46.711784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.711713 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" event={"ID":"2298e006-07b2-4516-84f3-1151da2a2991","Type":"ContainerDied","Data":"6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd"} Oct 31 03:56:46.711784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.711773 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl" event={"ID":"2298e006-07b2-4516-84f3-1151da2a2991","Type":"ContainerDied","Data":"602e54d217d334c66f16809e974a70857f027e32b9627c1fbcedc3d7f5da39d7"} Oct 31 03:56:46.711934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.711797 4934 scope.go:122] "RemoveContainer" containerID="6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd" Oct 31 03:56:46.713447 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.713422 4934 generic.go:356] "Generic (PLEG): container finished" podID="2a111075-2227-42a4-a864-54fe1fbc54a6" containerID="ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6" exitCode=0 Oct 31 03:56:46.713561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.713475 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" Oct 31 03:56:46.713561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.713543 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" event={"ID":"2a111075-2227-42a4-a864-54fe1fbc54a6","Type":"ContainerDied","Data":"ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6"} Oct 31 03:56:46.713693 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.713590 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh" event={"ID":"2a111075-2227-42a4-a864-54fe1fbc54a6","Type":"ContainerDied","Data":"eb5b3636607c1b0afad3772f11cef37a81dda490c7ee0fa734f0889013b28735"} Oct 31 03:56:46.724995 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.724982 4934 scope.go:122] "RemoveContainer" containerID="6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd" Oct 31 03:56:46.725324 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:56:46.725305 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd\": container with ID starting with 6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd not found: ID does not exist" containerID="6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd" Oct 31 03:56:46.725415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.725333 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd"} err="failed to get container status \"6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd\": rpc error: code = NotFound desc = could not find container \"6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd\": container with ID starting with 6f392ef76a23033becced89e75b8208b7c43b3f20752ef995b3cc78d647df8cd not found: ID does not exist" Oct 31 03:56:46.725415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.725349 4934 scope.go:122] "RemoveContainer" containerID="ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6" Oct 31 03:56:46.735260 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.735221 4934 scope.go:122] "RemoveContainer" containerID="ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6" Oct 31 03:56:46.735633 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:56:46.735606 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6\": container with ID starting with ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6 not found: ID does not exist" containerID="ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6" Oct 31 03:56:46.735746 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.735641 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6"} err="failed to get container status \"ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6\": rpc error: code = NotFound desc = could not find container \"ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6\": container with ID starting with ff91331b7ec7880bc78e6698e5201d5224f8fbe2610762aa61e3804d8c6f5ea6 not found: ID does not exist" Oct 31 03:56:46.787734 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.787706 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/a0c01170-981e-423d-8529-5ae0076a99de-kube-api-access-rfnnv\") pod \"test-curl-7165-ea87cd\" (UID: \"a0c01170-981e-423d-8529-5ae0076a99de\") " pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:46.848779 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.848759 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/a0c01170-981e-423d-8529-5ae0076a99de-kube-api-access-rfnnv\") pod \"test-curl-7165-ea87cd\" (UID: \"a0c01170-981e-423d-8529-5ae0076a99de\") " pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:46.877994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.877973 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl"] Oct 31 03:56:46.884550 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.884531 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-fnrcl"] Oct 31 03:56:46.885781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.885765 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:46.994069 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:46.994040 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh"] Oct 31 03:56:47.000473 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:47.000443 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-k9slh"] Oct 31 03:56:47.259915 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:47.259889 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7165-ea87cd"] Oct 31 03:56:47.262079 ip-10-0-138-238 kubenswrapper[4934]: W1031 03:56:47.262047 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0c01170_981e_423d_8529_5ae0076a99de.slice/crio-f6e7b61d368d1af6a27b4553c7af5d748e78705f33089b9635f3166a3f205c1d WatchSource:0}: Error finding container f6e7b61d368d1af6a27b4553c7af5d748e78705f33089b9635f3166a3f205c1d: Status 404 returned error can't find the container with id f6e7b61d368d1af6a27b4553c7af5d748e78705f33089b9635f3166a3f205c1d Oct 31 03:56:47.731734 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:47.731699 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7165-ea87cd" event={"ID":"a0c01170-981e-423d-8529-5ae0076a99de","Type":"ContainerStarted","Data":"ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd"} Oct 31 03:56:47.731929 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:47.731742 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7165-ea87cd" event={"ID":"a0c01170-981e-423d-8529-5ae0076a99de","Type":"ContainerStarted","Data":"f6e7b61d368d1af6a27b4553c7af5d748e78705f33089b9635f3166a3f205c1d"} Oct 31 03:56:47.854031 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:47.853972 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7165-ea87cd" podStartSLOduration=1.8539515309999999 podStartE2EDuration="1.853951531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:47.852481816 +0000 UTC m=+2241.171224442" watchObservedRunningTime="2026-10-31 03:56:47.853951531 +0000 UTC m=+2241.172694196" Oct 31 03:56:48.822059 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:48.822024 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2298e006-07b2-4516-84f3-1151da2a2991" path="/var/lib/kubelet/pods/2298e006-07b2-4516-84f3-1151da2a2991/volumes" Oct 31 03:56:48.822603 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:48.822587 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a111075-2227-42a4-a864-54fe1fbc54a6" path="/var/lib/kubelet/pods/2a111075-2227-42a4-a864-54fe1fbc54a6/volumes" Oct 31 03:56:49.165273 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:56:49.165170 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:56:49.445948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:49.445868 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7165-ea87cd"] Oct 31 03:56:49.451110 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:49.451077 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7165-ea87cd"] Oct 31 03:56:49.744608 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:49.744523 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7165-ea87cd" podUID="a0c01170-981e-423d-8529-5ae0076a99de" containerName="test-curl-7165-ea87cd" containerID="cri-o://ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd" gracePeriod=2 Oct 31 03:56:49.795872 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:56:49.795836 4934 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7165-ea87cd\" is forbidden: User \"system:node:ip-10-0-138-238.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-238.us-west-1.compute.internal' and this object" podUID="a0c01170-981e-423d-8529-5ae0076a99de" pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:51.995957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:51.995932 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:52.085193 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.085144 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c01170-981e-423d-8529-5ae0076a99de-kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/a0c01170-981e-423d-8529-5ae0076a99de-kube-api-access-rfnnv\") pod \"a0c01170-981e-423d-8529-5ae0076a99de\" (UID: \"a0c01170-981e-423d-8529-5ae0076a99de\") " Oct 31 03:56:52.088264 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.088222 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c01170-981e-423d-8529-5ae0076a99de-kube-api-access-rfnnv" pod "a0c01170-981e-423d-8529-5ae0076a99de" (UID: "a0c01170-981e-423d-8529-5ae0076a99de"). InnerVolumeSpecName "kube-api-access-rfnnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:52.187341 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.187288 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnnv\" (UniqueName: \"kubernetes.io/projected/a0c01170-981e-423d-8529-5ae0076a99de-kube-api-access-rfnnv\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.765656 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.765609 4934 generic.go:356] "Generic (PLEG): container finished" podID="a0c01170-981e-423d-8529-5ae0076a99de" containerID="ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd" exitCode=137 Oct 31 03:56:52.765863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.765670 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:56:52.765863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.765710 4934 scope.go:122] "RemoveContainer" containerID="ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd" Oct 31 03:56:52.775595 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.775577 4934 scope.go:122] "RemoveContainer" containerID="ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd" Oct 31 03:56:52.775899 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:56:52.775872 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd\": container with ID starting with ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd not found: ID does not exist" containerID="ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd" Oct 31 03:56:52.776019 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.775911 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd"} err="failed to get container status \"ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd\": rpc error: code = NotFound desc = could not find container \"ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd\": container with ID starting with ae195b2f29af384b283e720389afd627219d7530a6de7099a63542797c7c0dfd not found: ID does not exist" Oct 31 03:56:52.822135 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:56:52.822101 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c01170-981e-423d-8529-5ae0076a99de" path="/var/lib/kubelet/pods/a0c01170-981e-423d-8529-5ae0076a99de/volumes" Oct 31 03:56:52.829071 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:56:52.829049 4934 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7165-ea87cd\" is forbidden: User \"system:node:ip-10-0-138-238.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-238.us-west-1.compute.internal' and this object" podUID="a0c01170-981e-423d-8529-5ae0076a99de" pod="opendatahub/test-curl-7165-ea87cd" Oct 31 03:57:04.109907 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:04.109873 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7"] Oct 31 03:57:04.110604 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:04.110066 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" containerID="cri-o://ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50" gracePeriod=30 Oct 31 03:57:04.589253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:04.584394 4934 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:57:04.589253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:04.584473 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:57:05.885106 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:05.885077 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp"] Oct 31 03:57:05.885529 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:05.885291 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" podUID="4c9933ee-1180-4228-a595-440a790a7d20" containerName="maas-api" containerID="cri-o://9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907" gracePeriod=30 Oct 31 03:57:06.257423 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.257394 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:57:06.389021 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.388962 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls\") pod \"4c9933ee-1180-4228-a595-440a790a7d20\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " Oct 31 03:57:06.389214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.389047 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9933ee-1180-4228-a595-440a790a7d20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9933ee-1180-4228-a595-440a790a7d20-openshift-service-ca\") pod \"4c9933ee-1180-4228-a595-440a790a7d20\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " Oct 31 03:57:06.389214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.389106 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-metrics-certs\") pod \"4c9933ee-1180-4228-a595-440a790a7d20\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " Oct 31 03:57:06.389214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.389152 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9933ee-1180-4228-a595-440a790a7d20-kube-api-access-xjflv\" (UniqueName: \"kubernetes.io/projected/4c9933ee-1180-4228-a595-440a790a7d20-kube-api-access-xjflv\") pod \"4c9933ee-1180-4228-a595-440a790a7d20\" (UID: \"4c9933ee-1180-4228-a595-440a790a7d20\") " Oct 31 03:57:06.390851 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.390820 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9933ee-1180-4228-a595-440a790a7d20-openshift-service-ca" pod "4c9933ee-1180-4228-a595-440a790a7d20" (UID: "4c9933ee-1180-4228-a595-440a790a7d20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:06.398609 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.398559 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls" pod "4c9933ee-1180-4228-a595-440a790a7d20" (UID: "4c9933ee-1180-4228-a595-440a790a7d20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:06.398982 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.398960 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9933ee-1180-4228-a595-440a790a7d20-kube-api-access-xjflv" pod "4c9933ee-1180-4228-a595-440a790a7d20" (UID: "4c9933ee-1180-4228-a595-440a790a7d20"). InnerVolumeSpecName "kube-api-access-xjflv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:06.411043 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.411012 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-metrics-certs" pod "4c9933ee-1180-4228-a595-440a790a7d20" (UID: "4c9933ee-1180-4228-a595-440a790a7d20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:06.491435 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.491363 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:06.491435 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.491393 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9933ee-1180-4228-a595-440a790a7d20-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:06.491435 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.491403 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c9933ee-1180-4228-a595-440a790a7d20-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:06.491435 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.491412 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjflv\" (UniqueName: \"kubernetes.io/projected/4c9933ee-1180-4228-a595-440a790a7d20-kube-api-access-xjflv\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:06.899617 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.899585 4934 generic.go:356] "Generic (PLEG): container finished" podID="4c9933ee-1180-4228-a595-440a790a7d20" containerID="9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907" exitCode=0 Oct 31 03:57:06.900326 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.900212 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" event={"ID":"4c9933ee-1180-4228-a595-440a790a7d20","Type":"ContainerDied","Data":"9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907"} Oct 31 03:57:06.900551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.900301 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" event={"ID":"4c9933ee-1180-4228-a595-440a790a7d20","Type":"ContainerDied","Data":"cef98e05ea01aba04d8f27e8411ac27faddf24f328dc87fb7e17e6b478731000"} Oct 31 03:57:06.900810 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.900775 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp" Oct 31 03:57:06.902108 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.902093 4934 scope.go:122] "RemoveContainer" containerID="9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907" Oct 31 03:57:06.946967 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.946933 4934 scope.go:122] "RemoveContainer" containerID="9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907" Oct 31 03:57:06.958870 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:57:06.958757 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907\": container with ID starting with 9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907 not found: ID does not exist" containerID="9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907" Oct 31 03:57:06.958870 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:06.958799 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907"} err="failed to get container status \"9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907\": rpc error: code = NotFound desc = could not find container \"9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907\": container with ID starting with 9c2d118b4ba1e24ee839947ab2570b76ae6d3401f6882f6f3ce9e7d469cae907 not found: ID does not exist" Oct 31 03:57:07.068886 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:07.068841 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp"] Oct 31 03:57:07.118687 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:07.118616 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c43a6-84cdfd66cc-hnjnp"] Oct 31 03:57:08.822262 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:08.822217 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9933ee-1180-4228-a595-440a790a7d20" path="/var/lib/kubelet/pods/4c9933ee-1180-4228-a595-440a790a7d20/volumes" Oct 31 03:57:09.394312 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.394246 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:57:09.531980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.531951 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-certs\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532044 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-podinfo\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532150 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532105 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-credential-socket\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532184 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-kube-api-access-s5kpl\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-kube-api-access-s5kpl\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532264 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-socket\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532302 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532278 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-certs" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.532452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532320 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-data\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532358 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-envoy\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532407 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-token\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-token\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532434 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istiod-ca-cert\") pod \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\" (UID: \"2b2e050d-524e-4844-ba8c-9e4955f9ff7c\") " Oct 31 03:57:09.532647 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532489 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-credential-socket" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.532647 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532519 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-socket" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.532647 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532556 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-data" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.532930 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532904 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.533215 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532933 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.533215 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532948 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.533215 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532962 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.533215 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.532965 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istiod-ca-cert" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:09.534986 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.534953 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-envoy" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.534986 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.534978 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-podinfo" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:09.535558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.535528 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-token" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:09.537091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.537008 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-kube-api-access-s5kpl" pod "2b2e050d-524e-4844-ba8c-9e4955f9ff7c" (UID: "2b2e050d-524e-4844-ba8c-9e4955f9ff7c"). InnerVolumeSpecName "kube-api-access-s5kpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:09.634470 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.634444 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.634470 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.634469 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5kpl\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-kube-api-access-s5kpl\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.634616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.634479 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.634616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.634488 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.634616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.634496 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2e050d-524e-4844-ba8c-9e4955f9ff7c-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.921460 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.921427 4934 generic.go:356] "Generic (PLEG): container finished" podID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerID="ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50" exitCode=0 Oct 31 03:57:09.921921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.921473 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" Oct 31 03:57:09.921921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.921515 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" event={"ID":"2b2e050d-524e-4844-ba8c-9e4955f9ff7c","Type":"ContainerDied","Data":"ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50"} Oct 31 03:57:09.921921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.921576 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7" event={"ID":"2b2e050d-524e-4844-ba8c-9e4955f9ff7c","Type":"ContainerDied","Data":"dd32636a74d0f5eb227b58bc16f22a4fbbe2876b216e5bbc2d2128eb87d5e754"} Oct 31 03:57:09.921921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.921609 4934 scope.go:122] "RemoveContainer" containerID="ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50" Oct 31 03:57:09.932034 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.932020 4934 scope.go:122] "RemoveContainer" containerID="ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50" Oct 31 03:57:09.932309 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:57:09.932290 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50\": container with ID starting with ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50 not found: ID does not exist" containerID="ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50" Oct 31 03:57:09.932415 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:09.932317 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50"} err="failed to get container status \"ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50\": rpc error: code = NotFound desc = could not find container \"ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50\": container with ID starting with ef78b568aafb6bc90a9a3ca8a75a60aead31bd2714312ba4248eb389b9beae50 not found: ID does not exist" Oct 31 03:57:10.087852 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:10.087827 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7"] Oct 31 03:57:10.138891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:10.138864 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-9r8v7"] Oct 31 03:57:10.822073 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:10.822040 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" path="/var/lib/kubelet/pods/2b2e050d-524e-4844-ba8c-9e4955f9ff7c/volumes" Oct 31 03:57:14.869763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:14.869729 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj"] Oct 31 03:57:14.870299 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:14.869978 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" containerID="cri-o://e295677dcfb62cc253cebb6548202b20369a6ee45bad82ce52a95e3c6d61ce5d" gracePeriod=30 Oct 31 03:57:18.143700 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:18.143659 4934 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:57:18.144084 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:18.143733 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:57:19.165664 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:57:19.165618 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:57:20.086646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.086606 4934 generic.go:356] "Generic (PLEG): container finished" podID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerID="e295677dcfb62cc253cebb6548202b20369a6ee45bad82ce52a95e3c6d61ce5d" exitCode=0 Oct 31 03:57:20.086826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.086705 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" event={"ID":"520f796a-3221-4eaf-b27f-0b8c643476f1","Type":"ContainerDied","Data":"e295677dcfb62cc253cebb6548202b20369a6ee45bad82ce52a95e3c6d61ce5d"} Oct 31 03:57:20.135684 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.135629 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:57:20.255874 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.255847 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-certs\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.255911 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-podinfo\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.255950 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-credential-socket\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256005 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-envoy\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256136 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-certs" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:20.256354 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256275 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-credential-socket" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:20.256621 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256362 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520f796a-3221-4eaf-b27f-0b8c643476f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520f796a-3221-4eaf-b27f-0b8c643476f1-istiod-ca-cert\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256621 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256586 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-kube-api-access-czslf\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-kube-api-access-czslf\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256727 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256656 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520f796a-3221-4eaf-b27f-0b8c643476f1-istiod-ca-cert" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:20.256786 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256771 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-token\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-token\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256850 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256823 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-data\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.256944 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.256926 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-socket\") pod \"520f796a-3221-4eaf-b27f-0b8c643476f1\" (UID: \"520f796a-3221-4eaf-b27f-0b8c643476f1\") " Oct 31 03:57:20.257336 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.257151 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-data" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:20.257436 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.257162 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-socket" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:20.258142 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.258120 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.258296 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.258281 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.258399 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.258386 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520f796a-3221-4eaf-b27f-0b8c643476f1-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.258490 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.258477 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.258574 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.258562 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.259105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.259074 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-podinfo" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:20.259848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.259822 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-envoy" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:20.259994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.259974 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-kube-api-access-czslf" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "kube-api-access-czslf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:20.260055 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.260005 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-token" pod "520f796a-3221-4eaf-b27f-0b8c643476f1" (UID: "520f796a-3221-4eaf-b27f-0b8c643476f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:20.359780 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.359750 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.359780 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.359775 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.359780 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.359784 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czslf\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-kube-api-access-czslf\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:20.359972 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:20.359794 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520f796a-3221-4eaf-b27f-0b8c643476f1-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:21.093976 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:21.093891 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" Oct 31 03:57:21.094134 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:21.093894 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj" event={"ID":"520f796a-3221-4eaf-b27f-0b8c643476f1","Type":"ContainerDied","Data":"9fca959504a3422e42f63abd26d6aadfcef4b273c2c0d4f1dd7a5e14d20ac571"} Oct 31 03:57:21.094134 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:21.094038 4934 scope.go:122] "RemoveContainer" containerID="e295677dcfb62cc253cebb6548202b20369a6ee45bad82ce52a95e3c6d61ce5d" Oct 31 03:57:21.311163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:21.311132 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj"] Oct 31 03:57:21.362214 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:21.362127 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-t2dhj"] Oct 31 03:57:22.822018 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:22.821988 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" path="/var/lib/kubelet/pods/520f796a-3221-4eaf-b27f-0b8c643476f1/volumes" Oct 31 03:57:32.141249 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:32.141189 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb"] Oct 31 03:57:32.141697 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:32.141458 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" containerID="cri-o://809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8" gracePeriod=30 Oct 31 03:57:34.985888 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:34.985850 4934 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:57:34.986318 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:34.985918 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:57:37.424238 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.424197 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:57:37.502677 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.502644 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-socket\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.502833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.502700 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istiod-ca-cert\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.502833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.502808 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-envoy\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.502970 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.502846 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-token\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-token\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.502970 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.502880 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-credential-socket\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.503081 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.502970 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-data\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.503081 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503002 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-podinfo\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.503186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503099 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-certs\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.503186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503134 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-kube-api-access-bkgf5\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-kube-api-access-bkgf5\") pod \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\" (UID: \"f5e73b68-b1ec-41c2-81bc-160ad40bce81\") " Oct 31 03:57:37.503186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503165 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-credential-socket" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:37.503357 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503207 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istiod-ca-cert" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:37.503430 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503405 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-socket" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:37.503430 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503417 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-certs" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:37.503634 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503610 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.503755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503640 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.503755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503653 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.503755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503665 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.503755 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.503597 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-data" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:37.505953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.505927 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-podinfo" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:37.506050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.506010 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-kube-api-access-bkgf5" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "kube-api-access-bkgf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:37.506745 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.506713 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-token" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:37.507818 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.507779 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-envoy" pod "f5e73b68-b1ec-41c2-81bc-160ad40bce81" (UID: "f5e73b68-b1ec-41c2-81bc-160ad40bce81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:37.604718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.604673 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.604718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.604704 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.604718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.604713 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.604718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.604722 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e73b68-b1ec-41c2-81bc-160ad40bce81-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.605050 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:37.604732 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkgf5\" (UniqueName: \"kubernetes.io/projected/f5e73b68-b1ec-41c2-81bc-160ad40bce81-kube-api-access-bkgf5\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:38.255344 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.255291 4934 generic.go:356] "Generic (PLEG): container finished" podID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerID="809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8" exitCode=0 Oct 31 03:57:38.255344 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.255339 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" event={"ID":"f5e73b68-b1ec-41c2-81bc-160ad40bce81","Type":"ContainerDied","Data":"809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8"} Oct 31 03:57:38.255601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.255379 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" Oct 31 03:57:38.255601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.255398 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb" event={"ID":"f5e73b68-b1ec-41c2-81bc-160ad40bce81","Type":"ContainerDied","Data":"a2fa3c522fe126c3b2b808eea112faaf09ed69fb73b3ecadb339f4261237df92"} Oct 31 03:57:38.255601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.255423 4934 scope.go:122] "RemoveContainer" containerID="809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8" Oct 31 03:57:38.265900 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.265702 4934 scope.go:122] "RemoveContainer" containerID="809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8" Oct 31 03:57:38.266054 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:57:38.266035 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8\": container with ID starting with 809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8 not found: ID does not exist" containerID="809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8" Oct 31 03:57:38.266100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.266065 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8"} err="failed to get container status \"809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8\": rpc error: code = NotFound desc = could not find container \"809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8\": container with ID starting with 809d39c5e502d3f86b688b70d0db542401a542d60edbaaf652e2789a4a3970c8 not found: ID does not exist" Oct 31 03:57:38.471936 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.471901 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb"] Oct 31 03:57:38.522686 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.522620 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-jwprb"] Oct 31 03:57:38.821913 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:38.821884 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" path="/var/lib/kubelet/pods/f5e73b68-b1ec-41c2-81bc-160ad40bce81/volumes" Oct 31 03:57:45.050534 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.050505 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm"] Oct 31 03:57:45.050949 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.050741 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" podUID="ac2d9504-ef57-4231-bfa3-67ae4a8db56e" containerName="maas-api" containerID="cri-o://be4eba77266fab017c9ac0ab9dbf422bfc15c79f00ca641fc0da1445dcef83f4" gracePeriod=30 Oct 31 03:57:45.343937 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.343903 4934 generic.go:356] "Generic (PLEG): container finished" podID="ac2d9504-ef57-4231-bfa3-67ae4a8db56e" containerID="be4eba77266fab017c9ac0ab9dbf422bfc15c79f00ca641fc0da1445dcef83f4" exitCode=0 Oct 31 03:57:45.344163 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.343948 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" event={"ID":"ac2d9504-ef57-4231-bfa3-67ae4a8db56e","Type":"ContainerDied","Data":"be4eba77266fab017c9ac0ab9dbf422bfc15c79f00ca641fc0da1445dcef83f4"} Oct 31 03:57:45.371843 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.371822 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:57:45.525464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.521867 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-maas-api-tls\") pod \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " Oct 31 03:57:45.525464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.522036 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-openshift-service-ca\") pod \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " Oct 31 03:57:45.525464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.522079 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-metrics-certs\") pod \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " Oct 31 03:57:45.525464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.522162 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-kube-api-access-hbwbk\" (UniqueName: \"kubernetes.io/projected/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-kube-api-access-hbwbk\") pod \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\" (UID: \"ac2d9504-ef57-4231-bfa3-67ae4a8db56e\") " Oct 31 03:57:45.525464 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.523474 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-openshift-service-ca" pod "ac2d9504-ef57-4231-bfa3-67ae4a8db56e" (UID: "ac2d9504-ef57-4231-bfa3-67ae4a8db56e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:45.531848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.531815 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-metrics-certs" pod "ac2d9504-ef57-4231-bfa3-67ae4a8db56e" (UID: "ac2d9504-ef57-4231-bfa3-67ae4a8db56e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:45.532096 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.531947 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-kube-api-access-hbwbk" pod "ac2d9504-ef57-4231-bfa3-67ae4a8db56e" (UID: "ac2d9504-ef57-4231-bfa3-67ae4a8db56e"). InnerVolumeSpecName "kube-api-access-hbwbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:45.541688 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.541659 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-maas-api-tls" pod "ac2d9504-ef57-4231-bfa3-67ae4a8db56e" (UID: "ac2d9504-ef57-4231-bfa3-67ae4a8db56e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:45.625891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.625792 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:45.626268 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.626200 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:45.626432 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.626416 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwbk\" (UniqueName: \"kubernetes.io/projected/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-kube-api-access-hbwbk\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:45.632045 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:45.631975 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2d9504-ef57-4231-bfa3-67ae4a8db56e-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.351621 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:46.351591 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" Oct 31 03:57:46.352029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:46.351626 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm" event={"ID":"ac2d9504-ef57-4231-bfa3-67ae4a8db56e","Type":"ContainerDied","Data":"87a008628f162bafe95a0d5c6a884f48cc4071c6c82260745d6a5b0e0f258f8a"} Oct 31 03:57:46.352029 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:46.351689 4934 scope.go:122] "RemoveContainer" containerID="be4eba77266fab017c9ac0ab9dbf422bfc15c79f00ca641fc0da1445dcef83f4" Oct 31 03:57:46.517494 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:46.517461 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm"] Oct 31 03:57:46.567444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:46.567411 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84e130-74d5c87848-7kzmm"] Oct 31 03:57:46.824510 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:46.824475 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2d9504-ef57-4231-bfa3-67ae4a8db56e" path="/var/lib/kubelet/pods/ac2d9504-ef57-4231-bfa3-67ae4a8db56e/volumes" Oct 31 03:57:49.165483 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:57:49.165444 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:57:54.585113 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:54.585078 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x"] Oct 31 03:57:54.585611 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:54.585312 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" containerID="cri-o://95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f" gracePeriod=30 Oct 31 03:57:55.511692 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:55.511654 4934 patch_prober.go:29] interesting pod/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:57:55.511858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:55.511727 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:57:59.916393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:57:59.916370 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:58:00.038609 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038577 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25a196b3-a77a-414a-9995-b766598757aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a196b3-a77a-414a-9995-b766598757aa-istio-podinfo\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.038784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038625 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a196b3-a77a-414a-9995-b766598757aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a196b3-a77a-414a-9995-b766598757aa-istiod-ca-cert\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.038784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038674 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-data\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.038784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038723 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-certs\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.038784 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038784 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-socket\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.039016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038889 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-kube-api-access-dh9sf\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-kube-api-access-dh9sf\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.039016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038918 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-credential-socket\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.039016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038942 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-data" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:00.039016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.038976 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-istio-token\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-istio-token\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.039016 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039013 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-envoy\") pod \"25a196b3-a77a-414a-9995-b766598757aa\" (UID: \"25a196b3-a77a-414a-9995-b766598757aa\") " Oct 31 03:58:00.039298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039041 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-certs" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:00.039298 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039074 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a196b3-a77a-414a-9995-b766598757aa-istiod-ca-cert" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:00.039561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039369 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-socket" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:00.039561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039389 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-credential-socket" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:00.039561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039517 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a196b3-a77a-414a-9995-b766598757aa-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.039561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039531 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.039561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039541 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.039561 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039555 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.039864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.039567 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.041158 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.041135 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25a196b3-a77a-414a-9995-b766598757aa-istio-podinfo" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:00.041406 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.041218 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-kube-api-access-dh9sf" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "kube-api-access-dh9sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:00.041549 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.041436 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-istio-token" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:00.041751 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.041725 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-envoy" pod "25a196b3-a77a-414a-9995-b766598757aa" (UID: "25a196b3-a77a-414a-9995-b766598757aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:00.140655 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.140578 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9sf\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-kube-api-access-dh9sf\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.140655 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.140605 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a196b3-a77a-414a-9995-b766598757aa-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.140655 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.140614 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a196b3-a77a-414a-9995-b766598757aa-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.140655 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.140622 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a196b3-a77a-414a-9995-b766598757aa-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:00.456922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.456837 4934 generic.go:356] "Generic (PLEG): container finished" podID="25a196b3-a77a-414a-9995-b766598757aa" containerID="95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f" exitCode=0 Oct 31 03:58:00.456922 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.456893 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" event={"ID":"25a196b3-a77a-414a-9995-b766598757aa","Type":"ContainerDied","Data":"95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f"} Oct 31 03:58:00.457105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.456946 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" event={"ID":"25a196b3-a77a-414a-9995-b766598757aa","Type":"ContainerDied","Data":"54478e7be7dd3ecc84ec8b19da9504b1be5567b0096335fb93c7c8978bd3deda"} Oct 31 03:58:00.457105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.456957 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x" Oct 31 03:58:00.457105 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.456976 4934 scope.go:122] "RemoveContainer" containerID="95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f" Oct 31 03:58:00.467941 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.467924 4934 scope.go:122] "RemoveContainer" containerID="0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3" Oct 31 03:58:00.476455 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.476440 4934 scope.go:122] "RemoveContainer" containerID="95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f" Oct 31 03:58:00.476731 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:00.476715 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f\": container with ID starting with 95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f not found: ID does not exist" containerID="95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f" Oct 31 03:58:00.476777 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.476737 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f"} err="failed to get container status \"95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f\": rpc error: code = NotFound desc = could not find container \"95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f\": container with ID starting with 95d90860b80a6e5a8944780b79ba5b8ba3db3e28770656e4a92aaf5f2cd9936f not found: ID does not exist" Oct 31 03:58:00.476777 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.476754 4934 scope.go:122] "RemoveContainer" containerID="0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3" Oct 31 03:58:00.477010 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:00.476990 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3\": container with ID starting with 0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3 not found: ID does not exist" containerID="0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3" Oct 31 03:58:00.477062 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.477017 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3"} err="failed to get container status \"0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3\": rpc error: code = NotFound desc = could not find container \"0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3\": container with ID starting with 0638c8a7edf6c6f9648b91fd5b3c44bc3aae87c4ed4817306a6d42e00e8217f3 not found: ID does not exist" Oct 31 03:58:00.622569 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.622537 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x"] Oct 31 03:58:00.673387 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.673360 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-84e130-openshift-default-59d5b8c787-blt7x"] Oct 31 03:58:00.823872 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:00.823838 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a196b3-a77a-414a-9995-b766598757aa" path="/var/lib/kubelet/pods/25a196b3-a77a-414a-9995-b766598757aa/volumes" Oct 31 03:58:19.163384 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:19.163349 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:58:26.480370 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.480337 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs"] Oct 31 03:58:26.480776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.480542 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" podUID="f05f22f9-4f34-4f85-836b-6d895a800855" containerName="maas-api" containerID="cri-o://21f5269d7f08d31074ef20cd782642d5638ca269fda02a1cd528033dcd082703" gracePeriod=30 Oct 31 03:58:26.621196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.621166 4934 generic.go:356] "Generic (PLEG): container finished" podID="f05f22f9-4f34-4f85-836b-6d895a800855" containerID="21f5269d7f08d31074ef20cd782642d5638ca269fda02a1cd528033dcd082703" exitCode=0 Oct 31 03:58:26.621383 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.621270 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" event={"ID":"f05f22f9-4f34-4f85-836b-6d895a800855","Type":"ContainerDied","Data":"21f5269d7f08d31074ef20cd782642d5638ca269fda02a1cd528033dcd082703"} Oct 31 03:58:26.811022 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.810995 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:58:26.878412 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.864102 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05f22f9-4f34-4f85-836b-6d895a800855-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f22f9-4f34-4f85-836b-6d895a800855-openshift-service-ca\") pod \"f05f22f9-4f34-4f85-836b-6d895a800855\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " Oct 31 03:58:26.878412 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.864247 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls\") pod \"f05f22f9-4f34-4f85-836b-6d895a800855\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " Oct 31 03:58:26.878412 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.864381 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05f22f9-4f34-4f85-836b-6d895a800855-kube-api-access-nn6t7\" (UniqueName: \"kubernetes.io/projected/f05f22f9-4f34-4f85-836b-6d895a800855-kube-api-access-nn6t7\") pod \"f05f22f9-4f34-4f85-836b-6d895a800855\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " Oct 31 03:58:26.878412 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.864417 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-metrics-certs\") pod \"f05f22f9-4f34-4f85-836b-6d895a800855\" (UID: \"f05f22f9-4f34-4f85-836b-6d895a800855\") " Oct 31 03:58:26.878953 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.878919 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05f22f9-4f34-4f85-836b-6d895a800855-openshift-service-ca" pod "f05f22f9-4f34-4f85-836b-6d895a800855" (UID: "f05f22f9-4f34-4f85-836b-6d895a800855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:26.889836 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.889764 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls" pod "f05f22f9-4f34-4f85-836b-6d895a800855" (UID: "f05f22f9-4f34-4f85-836b-6d895a800855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:26.889836 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.889815 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05f22f9-4f34-4f85-836b-6d895a800855-kube-api-access-nn6t7" pod "f05f22f9-4f34-4f85-836b-6d895a800855" (UID: "f05f22f9-4f34-4f85-836b-6d895a800855"). InnerVolumeSpecName "kube-api-access-nn6t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:26.898311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.897388 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-metrics-certs" pod "f05f22f9-4f34-4f85-836b-6d895a800855" (UID: "f05f22f9-4f34-4f85-836b-6d895a800855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:26.966074 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.966039 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn6t7\" (UniqueName: \"kubernetes.io/projected/f05f22f9-4f34-4f85-836b-6d895a800855-kube-api-access-nn6t7\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:26.966074 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.966070 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:26.966324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.966083 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f22f9-4f34-4f85-836b-6d895a800855-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:26.966324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:26.966096 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f22f9-4f34-4f85-836b-6d895a800855-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:27.628101 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:27.628067 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" Oct 31 03:58:27.628538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:27.628072 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs" event={"ID":"f05f22f9-4f34-4f85-836b-6d895a800855","Type":"ContainerDied","Data":"05c69a3355f159b62cfc11be16360054c9bece1c420e857133ba93d0979ed6a2"} Oct 31 03:58:27.628538 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:27.628253 4934 scope.go:122] "RemoveContainer" containerID="21f5269d7f08d31074ef20cd782642d5638ca269fda02a1cd528033dcd082703" Oct 31 03:58:27.793895 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:27.793866 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs"] Oct 31 03:58:27.844285 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:27.844254 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1783b3-5594f4fb9-bw4fs"] Oct 31 03:58:28.821609 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:28.821577 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05f22f9-4f34-4f85-836b-6d895a800855" path="/var/lib/kubelet/pods/f05f22f9-4f34-4f85-836b-6d895a800855/volumes" Oct 31 03:58:32.073887 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:32.073844 4934 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": read tcp 10.132.0.2:60380->10.132.0.95:15021: read: connection reset by peer" start-of-body= Oct 31 03:58:32.074378 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:32.073907 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": read tcp 10.132.0.2:60380->10.132.0.95:15021: read: connection reset by peer" Oct 31 03:58:32.782183 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:32.782124 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv"] Oct 31 03:58:32.782501 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:32.782444 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" podUID="840e5980-4118-4f8d-988b-2920003c251c" containerName="maas-api" containerID="cri-o://2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6" gracePeriod=30 Oct 31 03:58:33.100131 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.100106 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:58:33.249527 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.249491 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840e5980-4118-4f8d-988b-2920003c251c-kube-api-access-pj4d9\" (UniqueName: \"kubernetes.io/projected/840e5980-4118-4f8d-988b-2920003c251c-kube-api-access-pj4d9\") pod \"840e5980-4118-4f8d-988b-2920003c251c\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " Oct 31 03:58:33.249724 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.249622 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-metrics-certs\") pod \"840e5980-4118-4f8d-988b-2920003c251c\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " Oct 31 03:58:33.249724 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.249659 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/840e5980-4118-4f8d-988b-2920003c251c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840e5980-4118-4f8d-988b-2920003c251c-openshift-service-ca\") pod \"840e5980-4118-4f8d-988b-2920003c251c\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " Oct 31 03:58:33.249724 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.249687 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls\") pod \"840e5980-4118-4f8d-988b-2920003c251c\" (UID: \"840e5980-4118-4f8d-988b-2920003c251c\") " Oct 31 03:58:33.250549 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.250520 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840e5980-4118-4f8d-988b-2920003c251c-openshift-service-ca" pod "840e5980-4118-4f8d-988b-2920003c251c" (UID: "840e5980-4118-4f8d-988b-2920003c251c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:33.252663 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.252641 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls" pod "840e5980-4118-4f8d-988b-2920003c251c" (UID: "840e5980-4118-4f8d-988b-2920003c251c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:33.252806 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.252665 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840e5980-4118-4f8d-988b-2920003c251c-kube-api-access-pj4d9" pod "840e5980-4118-4f8d-988b-2920003c251c" (UID: "840e5980-4118-4f8d-988b-2920003c251c"). InnerVolumeSpecName "kube-api-access-pj4d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:33.252987 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.252960 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-metrics-certs" pod "840e5980-4118-4f8d-988b-2920003c251c" (UID: "840e5980-4118-4f8d-988b-2920003c251c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:33.352117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.351964 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4d9\" (UniqueName: \"kubernetes.io/projected/840e5980-4118-4f8d-988b-2920003c251c-kube-api-access-pj4d9\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.352117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.352007 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.352117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.352022 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840e5980-4118-4f8d-988b-2920003c251c-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.352117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.352038 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840e5980-4118-4f8d-988b-2920003c251c-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.666487 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.666404 4934 generic.go:356] "Generic (PLEG): container finished" podID="840e5980-4118-4f8d-988b-2920003c251c" containerID="2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6" exitCode=0 Oct 31 03:58:33.666487 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.666466 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" Oct 31 03:58:33.666670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.666489 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" event={"ID":"840e5980-4118-4f8d-988b-2920003c251c","Type":"ContainerDied","Data":"2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6"} Oct 31 03:58:33.666670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.666542 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv" event={"ID":"840e5980-4118-4f8d-988b-2920003c251c","Type":"ContainerDied","Data":"bc72770763e49ce2ae557bece25cdbf157c41799272c87002da4265914958a57"} Oct 31 03:58:33.666670 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.666565 4934 scope.go:122] "RemoveContainer" containerID="2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6" Oct 31 03:58:33.676957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.676862 4934 scope.go:122] "RemoveContainer" containerID="2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6" Oct 31 03:58:33.677311 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:33.677212 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6\": container with ID starting with 2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6 not found: ID does not exist" containerID="2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6" Oct 31 03:58:33.677311 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.677277 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6"} err="failed to get container status \"2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6\": rpc error: code = NotFound desc = could not find container \"2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6\": container with ID starting with 2ae55dc0d395bb104dbfc7c475d462947694c2d94477281e4ff64e03b82c53b6 not found: ID does not exist" Oct 31 03:58:33.853879 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.853843 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv"] Oct 31 03:58:33.859609 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:33.859583 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e02dd5-6f97d86d98-c2nmv"] Oct 31 03:58:34.822616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:34.822583 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="840e5980-4118-4f8d-988b-2920003c251c" path="/var/lib/kubelet/pods/840e5980-4118-4f8d-988b-2920003c251c/volumes" Oct 31 03:58:35.900803 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:35.900771 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57"] Oct 31 03:58:35.901218 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:35.900958 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" containerID="cri-o://6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6" gracePeriod=30 Oct 31 03:58:36.668674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:36.668618 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb"] Oct 31 03:58:36.668905 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:36.668873 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" containerID="cri-o://d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce" gracePeriod=30 Oct 31 03:58:36.671362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:36.671337 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb"] Oct 31 03:58:36.671578 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:36.671555 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" containerID="cri-o://0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8" gracePeriod=30 Oct 31 03:58:36.717128 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:36.717099 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg"] Oct 31 03:58:36.717388 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:36.717365 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" containerID="cri-o://b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138" gracePeriod=30 Oct 31 03:58:41.166730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.166673 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:58:41.308281 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.308247 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s"] Oct 31 03:58:41.347975 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.347946 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-socket\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348111 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348007 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-token\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-token\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348111 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348096 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-data\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348180 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348168 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-credential-socket\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348271 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348252 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-socket" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:41.348315 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348287 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-certs\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348362 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348318 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-podinfo\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348406 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348353 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-envoy\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348457 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348402 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istiod-ca-cert\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348457 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348419 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-data" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:41.348957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348448 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-credential-socket" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:41.348957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348481 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-kube-api-access-7svf4\") pod \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\" (UID: \"e2375106-1e2f-44a8-a7bf-dda13216fe2a\") " Oct 31 03:58:41.348957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348561 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-certs" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:41.348957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348835 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istiod-ca-cert" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:41.348957 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348948 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.349166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348965 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.349166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348979 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.349166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.348993 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.349166 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.349006 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.350662 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.350639 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-kube-api-access-7svf4" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "kube-api-access-7svf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:41.350771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.350700 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-envoy" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:41.350823 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.350779 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-token" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:41.350886 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.350871 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-podinfo" pod "e2375106-1e2f-44a8-a7bf-dda13216fe2a" (UID: "e2375106-1e2f-44a8-a7bf-dda13216fe2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:41.450558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.450484 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.450558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.450514 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.450558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.450525 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-kube-api-access-7svf4\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.450558 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.450535 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2375106-1e2f-44a8-a7bf-dda13216fe2a-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:41.717338 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.717307 4934 generic.go:356] "Generic (PLEG): container finished" podID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerID="6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6" exitCode=0 Oct 31 03:58:41.717461 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.717366 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" Oct 31 03:58:41.717461 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.717392 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" event={"ID":"e2375106-1e2f-44a8-a7bf-dda13216fe2a","Type":"ContainerDied","Data":"6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6"} Oct 31 03:58:41.717567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.717465 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57" event={"ID":"e2375106-1e2f-44a8-a7bf-dda13216fe2a","Type":"ContainerDied","Data":"95a2d3980194874372f2dcfc7d56b42853962719581ba9ec15eef22ad50a7d09"} Oct 31 03:58:41.717567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.717481 4934 scope.go:122] "RemoveContainer" containerID="6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6" Oct 31 03:58:41.822921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.822898 4934 scope.go:122] "RemoveContainer" containerID="6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6" Oct 31 03:58:41.823335 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:41.823314 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6\": container with ID starting with 6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6 not found: ID does not exist" containerID="6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6" Oct 31 03:58:41.823428 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.823347 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6"} err="failed to get container status \"6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6\": rpc error: code = NotFound desc = could not find container \"6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6\": container with ID starting with 6131b10c206d1bbbd9ef167349952b9ce851976441571986c9ffd57161a9e4f6 not found: ID does not exist" Oct 31 03:58:41.933595 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.933564 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57"] Oct 31 03:58:42.002100 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:41.993461 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-wzz57"] Oct 31 03:58:42.013027 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.013006 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:58:42.073687 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.073660 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:58:42.077191 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.077169 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:58:42.158414 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158386 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-data\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158564 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158422 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-socket\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158564 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158454 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-certs\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158568 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istiod-ca-cert\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158596 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-envoy\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158640 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-podinfo\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158696 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158680 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-data" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.158910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158693 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-token\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-token\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158799 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-credential-socket\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.158910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.158869 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-kube-api-access-4gvpn\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-kube-api-access-4gvpn\") pod \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\" (UID: \"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8\") " Oct 31 03:58:42.160787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.159201 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-credential-socket" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.160787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.159519 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.160787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.159540 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.160787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.159773 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-certs" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.160787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.159984 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-socket" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.161549 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.161528 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istiod-ca-cert" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:42.162070 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.162032 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-token" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.162193 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.162166 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-kube-api-access-4gvpn" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "kube-api-access-4gvpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.162398 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.162374 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-podinfo" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:42.163350 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.163330 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-envoy" pod "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" (UID: "fe8088fc-01d4-4ce9-85f4-32fd06caf3f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260147 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-data\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260199 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istiod-ca-cert\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260267 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-socket\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260318 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-podinfo\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260350 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-data\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260389 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-certs\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260433 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-certs\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260490 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-credential-socket\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260534 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-token\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-token\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260562 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-istio-token\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-istio-token\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260612 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a06443ca-871f-413c-885f-200727268117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06443ca-871f-413c-885f-200727268117-istio-podinfo\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260641 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-envoy\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260682 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-socket\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260727 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-kube-api-access-5qhrz\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-kube-api-access-5qhrz\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260784 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-credential-socket\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.260980 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-kube-api-access-2c9cc\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-kube-api-access-2c9cc\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261040 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-envoy\") pod \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\" (UID: \"f6b7182b-43bf-4208-8836-ce6eb34a5a14\") " Oct 31 03:58:42.261551 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261080 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06443ca-871f-413c-885f-200727268117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06443ca-871f-413c-885f-200727268117-istiod-ca-cert\") pod \"a06443ca-871f-413c-885f-200727268117\" (UID: \"a06443ca-871f-413c-885f-200727268117\") " Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261568 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261587 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261601 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261615 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261627 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvpn\" (UniqueName: \"kubernetes.io/projected/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-kube-api-access-4gvpn\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261640 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.261653 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.262349 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06443ca-871f-413c-885f-200727268117-istiod-ca-cert" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:42.262948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.262934 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-socket" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.265065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.264263 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-data" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.265065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.264670 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-socket" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.265065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.264858 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-certs" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.265065 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.264889 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-certs" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.265356 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.265038 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-data" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.265356 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.265111 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-credential-socket" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.268198 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.268167 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-kube-api-access-2c9cc" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "kube-api-access-2c9cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.268857 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.268830 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-podinfo" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:42.270674 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.270649 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-istio-token" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.270775 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.270714 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-kube-api-access-5qhrz" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "kube-api-access-5qhrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.283626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.271059 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-envoy" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.283626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.271345 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-envoy" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.283626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.271355 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a06443ca-871f-413c-885f-200727268117-istio-podinfo" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:42.283626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.271390 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istiod-ca-cert" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:42.283626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.271570 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-credential-socket" pod "a06443ca-871f-413c-885f-200727268117" (UID: "a06443ca-871f-413c-885f-200727268117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.283626 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.272136 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-token" pod "f6b7182b-43bf-4208-8836-ce6eb34a5a14" (UID: "f6b7182b-43bf-4208-8836-ce6eb34a5a14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.362679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362641 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.362679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362668 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.362679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362677 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.362679 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362686 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06443ca-871f-413c-885f-200727268117-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362694 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362703 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362717 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qhrz\" (UniqueName: \"kubernetes.io/projected/a06443ca-871f-413c-885f-200727268117-kube-api-access-5qhrz\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362731 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362740 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c9cc\" (UniqueName: \"kubernetes.io/projected/f6b7182b-43bf-4208-8836-ce6eb34a5a14-kube-api-access-2c9cc\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362749 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362759 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06443ca-871f-413c-885f-200727268117-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362768 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362776 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362784 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362799 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b7182b-43bf-4208-8836-ce6eb34a5a14-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362808 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362817 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06443ca-871f-413c-885f-200727268117-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.363017 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.362825 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b7182b-43bf-4208-8836-ce6eb34a5a14-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.724698 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.724651 4934 generic.go:356] "Generic (PLEG): container finished" podID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerID="d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce" exitCode=0 Oct 31 03:58:42.724904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.724706 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" Oct 31 03:58:42.724904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.724739 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" event={"ID":"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8","Type":"ContainerDied","Data":"d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce"} Oct 31 03:58:42.724904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.724797 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb" event={"ID":"fe8088fc-01d4-4ce9-85f4-32fd06caf3f8","Type":"ContainerDied","Data":"a5e30b422a918db01ba7f6c75c05450c73361a5594cc546fc71dba259b88a165"} Oct 31 03:58:42.724904 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.724825 4934 scope.go:122] "RemoveContainer" containerID="d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce" Oct 31 03:58:42.726343 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.726319 4934 generic.go:356] "Generic (PLEG): container finished" podID="a06443ca-871f-413c-885f-200727268117" containerID="0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8" exitCode=0 Oct 31 03:58:42.726469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.726365 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" Oct 31 03:58:42.726469 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.726428 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" event={"ID":"a06443ca-871f-413c-885f-200727268117","Type":"ContainerDied","Data":"0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8"} Oct 31 03:58:42.726587 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.726504 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb" event={"ID":"a06443ca-871f-413c-885f-200727268117","Type":"ContainerDied","Data":"386105d8bd98f7da37f292a377858e8cba6a5ef6c6a10ea7b2db36cabe5a3c69"} Oct 31 03:58:42.727963 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.727939 4934 generic.go:356] "Generic (PLEG): container finished" podID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerID="b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138" exitCode=0 Oct 31 03:58:42.728061 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.727980 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" event={"ID":"f6b7182b-43bf-4208-8836-ce6eb34a5a14","Type":"ContainerDied","Data":"b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138"} Oct 31 03:58:42.728061 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.727985 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" Oct 31 03:58:42.728061 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.728013 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg" event={"ID":"f6b7182b-43bf-4208-8836-ce6eb34a5a14","Type":"ContainerDied","Data":"d1fc097e7bd058acee0474c6bb845384620d8e2ba72175b360c5ca6431cf6912"} Oct 31 03:58:42.737064 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.737022 4934 scope.go:122] "RemoveContainer" containerID="d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce" Oct 31 03:58:42.737362 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:42.737337 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce\": container with ID starting with d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce not found: ID does not exist" containerID="d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce" Oct 31 03:58:42.737474 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.737392 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce"} err="failed to get container status \"d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce\": rpc error: code = NotFound desc = could not find container \"d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce\": container with ID starting with d6bf3ca17e793de74069accf3ecd07b85eabaa693e30dda7d12fe63bcac4e9ce not found: ID does not exist" Oct 31 03:58:42.737474 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.737415 4934 scope.go:122] "RemoveContainer" containerID="0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8" Oct 31 03:58:42.747061 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.747047 4934 scope.go:122] "RemoveContainer" containerID="0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8" Oct 31 03:58:42.747741 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:42.747721 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8\": container with ID starting with 0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8 not found: ID does not exist" containerID="0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8" Oct 31 03:58:42.747812 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.747751 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8"} err="failed to get container status \"0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8\": rpc error: code = NotFound desc = could not find container \"0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8\": container with ID starting with 0d9bb671f74e9ff1d56870ced058cb13bb652f77d20af93abb9f2ae6a0feffb8 not found: ID does not exist" Oct 31 03:58:42.747812 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.747768 4934 scope.go:122] "RemoveContainer" containerID="b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138" Oct 31 03:58:42.757013 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.756996 4934 scope.go:122] "RemoveContainer" containerID="b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138" Oct 31 03:58:42.757332 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:42.757313 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138\": container with ID starting with b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138 not found: ID does not exist" containerID="b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138" Oct 31 03:58:42.757410 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.757340 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138"} err="failed to get container status \"b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138\": rpc error: code = NotFound desc = could not find container \"b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138\": container with ID starting with b95b01a62503fafbf024ef34d63f29354b745218bb7a5633cb2e4f4d851c6138 not found: ID does not exist" Oct 31 03:58:42.821961 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.821919 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" path="/var/lib/kubelet/pods/e2375106-1e2f-44a8-a7bf-dda13216fe2a/volumes" Oct 31 03:58:42.890781 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.890742 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb"] Oct 31 03:58:42.940402 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:42.940378 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-4tbtb"] Oct 31 03:58:43.014279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:43.006658 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb"] Oct 31 03:58:43.014279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:43.011489 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-nmlzb"] Oct 31 03:58:43.018615 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:43.018592 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj"] Oct 31 03:58:43.018866 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:43.018838 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" containerID="cri-o://cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584" gracePeriod=30 Oct 31 03:58:43.181910 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:43.181866 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg"] Oct 31 03:58:43.191914 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:43.191454 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-8bgbg"] Oct 31 03:58:44.822517 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:44.822487 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06443ca-871f-413c-885f-200727268117" path="/var/lib/kubelet/pods/a06443ca-871f-413c-885f-200727268117/volumes" Oct 31 03:58:44.822928 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:44.822914 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" path="/var/lib/kubelet/pods/f6b7182b-43bf-4208-8836-ce6eb34a5a14/volumes" Oct 31 03:58:44.823364 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:44.823351 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" path="/var/lib/kubelet/pods/fe8088fc-01d4-4ce9-85f4-32fd06caf3f8/volumes" Oct 31 03:58:48.287384 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.287329 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:58:48.335694 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335668 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-kube-api-access-dkhkd\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-kube-api-access-dkhkd\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.335847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335704 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-certs\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.335847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335761 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-credential-socket\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.335847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335792 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istiod-ca-cert\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.335847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335823 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-socket\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.335847 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335845 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-podinfo\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.336155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335885 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-token\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-token\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.336155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335949 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-data\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.336155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.335987 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-envoy\") pod \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\" (UID: \"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0\") " Oct 31 03:58:48.336155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336099 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-socket" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:48.336155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336116 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-certs" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:48.336590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336198 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istiod-ca-cert" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:48.336590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336360 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-data" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:48.336590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336448 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-credential-socket" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:48.336590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336513 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.336590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336528 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.336590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336538 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.336590 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.336553 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.338037 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.338008 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-kube-api-access-dkhkd" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "kube-api-access-dkhkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:48.338252 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.338211 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-token" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:48.338678 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.338653 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-podinfo" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:48.338734 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.338651 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-envoy" pod "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" (UID: "d106d91d-2fbd-4dbb-8b6f-5601d33d95b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:48.438013 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.437982 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.438013 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.438006 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.438013 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.438016 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhkd\" (UniqueName: \"kubernetes.io/projected/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-kube-api-access-dkhkd\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.438211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.438026 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.438211 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.438035 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.529844 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.529812 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b"] Oct 31 03:58:48.535479 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.535450 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc"] Oct 31 03:58:48.554705 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.554630 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g"] Oct 31 03:58:48.771193 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.771162 4934 generic.go:356] "Generic (PLEG): container finished" podID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerID="cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584" exitCode=0 Oct 31 03:58:48.771405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.771207 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" Oct 31 03:58:48.771405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.771256 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" event={"ID":"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0","Type":"ContainerDied","Data":"cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584"} Oct 31 03:58:48.771405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.771336 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj" event={"ID":"d106d91d-2fbd-4dbb-8b6f-5601d33d95b0","Type":"ContainerDied","Data":"5fc99da8f68e6df22f5e037d4f7734529747db45bb663302538ff2a7621ac06d"} Oct 31 03:58:48.771405 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.771366 4934 scope.go:122] "RemoveContainer" containerID="cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584" Oct 31 03:58:48.783317 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.783283 4934 scope.go:122] "RemoveContainer" containerID="3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64" Oct 31 03:58:48.794182 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.794136 4934 scope.go:122] "RemoveContainer" containerID="cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584" Oct 31 03:58:48.794510 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:48.794491 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584\": container with ID starting with cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584 not found: ID does not exist" containerID="cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584" Oct 31 03:58:48.794585 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.794517 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584"} err="failed to get container status \"cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584\": rpc error: code = NotFound desc = could not find container \"cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584\": container with ID starting with cb8418f931dd1d0587dabb37298634abbfaecf73497b4beba911fdefd1f33584 not found: ID does not exist" Oct 31 03:58:48.794585 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.794533 4934 scope.go:122] "RemoveContainer" containerID="3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64" Oct 31 03:58:48.794799 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:48.794779 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64\": container with ID starting with 3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64 not found: ID does not exist" containerID="3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64" Oct 31 03:58:48.794858 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.794813 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64"} err="failed to get container status \"3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64\": rpc error: code = NotFound desc = could not find container \"3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64\": container with ID starting with 3c8ddbcf1fac5d743d78d8fbbc789e75b2ce6bd9ba6e4e9530f36af8b209bc64 not found: ID does not exist" Oct 31 03:58:48.999287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:48.999250 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj"] Oct 31 03:58:49.003205 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:49.003180 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e02dd5-openshift-default-b796787cf-tqspj"] Oct 31 03:58:49.170927 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:49.170881 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:58:50.821718 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:50.821683 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" path="/var/lib/kubelet/pods/d106d91d-2fbd-4dbb-8b6f-5601d33d95b0/volumes" Oct 31 03:58:56.328614 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.328558 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" containerID="cri-o://0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb" gracePeriod=45 Oct 31 03:58:56.584618 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.584566 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:58:56.630807 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.630785 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-model-cache\") pod \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " Oct 31 03:58:56.630906 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.630859 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-dshm\") pod \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " Oct 31 03:58:56.630973 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.630958 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-home\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-home\") pod \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " Oct 31 03:58:56.631035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.630995 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a4cda6-f9c8-441c-ad18-c335c59fb134-kube-api-access-fh4rj\" (UniqueName: \"kubernetes.io/projected/d3a4cda6-f9c8-441c-ad18-c335c59fb134-kube-api-access-fh4rj\") pod \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " Oct 31 03:58:56.631091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.631043 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-model-cache" pod "d3a4cda6-f9c8-441c-ad18-c335c59fb134" (UID: "d3a4cda6-f9c8-441c-ad18-c335c59fb134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.631091 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.631057 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tls-certs\") pod \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " Oct 31 03:58:56.631201 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.631134 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tmp-dir\") pod \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\" (UID: \"d3a4cda6-f9c8-441c-ad18-c335c59fb134\") " Oct 31 03:58:56.631312 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.631198 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-home" pod "d3a4cda6-f9c8-441c-ad18-c335c59fb134" (UID: "d3a4cda6-f9c8-441c-ad18-c335c59fb134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.631515 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.631470 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tmp-dir" pod "d3a4cda6-f9c8-441c-ad18-c335c59fb134" (UID: "d3a4cda6-f9c8-441c-ad18-c335c59fb134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.631612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.631548 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.631612 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.631575 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.633324 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.633303 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a4cda6-f9c8-441c-ad18-c335c59fb134-kube-api-access-fh4rj" pod "d3a4cda6-f9c8-441c-ad18-c335c59fb134" (UID: "d3a4cda6-f9c8-441c-ad18-c335c59fb134"). InnerVolumeSpecName "kube-api-access-fh4rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:56.633408 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.633299 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-dshm" pod "d3a4cda6-f9c8-441c-ad18-c335c59fb134" (UID: "d3a4cda6-f9c8-441c-ad18-c335c59fb134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.633408 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.633334 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tls-certs" pod "d3a4cda6-f9c8-441c-ad18-c335c59fb134" (UID: "d3a4cda6-f9c8-441c-ad18-c335c59fb134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:56.732639 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.732608 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4rj\" (UniqueName: \"kubernetes.io/projected/d3a4cda6-f9c8-441c-ad18-c335c59fb134-kube-api-access-fh4rj\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.732639 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.732641 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.732856 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.732656 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.732856 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.732669 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a4cda6-f9c8-441c-ad18-c335c59fb134-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.821884 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.821853 4934 generic.go:356] "Generic (PLEG): container finished" podID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerID="0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb" exitCode=0 Oct 31 03:58:56.822043 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.821958 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" Oct 31 03:58:56.822086 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.821954 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" event={"ID":"d3a4cda6-f9c8-441c-ad18-c335c59fb134","Type":"ContainerDied","Data":"0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb"} Oct 31 03:58:56.822129 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.822081 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s" event={"ID":"d3a4cda6-f9c8-441c-ad18-c335c59fb134","Type":"ContainerDied","Data":"c1d2d01a6962bf0cb0d400f524014269b0dab3a065e38edbc59305ec1b5bd82c"} Oct 31 03:58:56.822129 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.822102 4934 scope.go:122] "RemoveContainer" containerID="0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb" Oct 31 03:58:56.831452 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.831431 4934 scope.go:122] "RemoveContainer" containerID="0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb" Oct 31 03:58:56.831706 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:58:56.831688 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb\": container with ID starting with 0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb not found: ID does not exist" containerID="0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb" Oct 31 03:58:56.831762 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.831714 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb"} err="failed to get container status \"0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb\": rpc error: code = NotFound desc = could not find container \"0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb\": container with ID starting with 0e37dc95560fdf01e6f10976654d7743214fcd98ca47e7c4ae3a604d9d355cbb not found: ID does not exist" Oct 31 03:58:56.986771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.986696 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s"] Oct 31 03:58:56.991161 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:56.991135 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-w6-b63425-kserve-56c46df849-vtl8s"] Oct 31 03:58:58.823446 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:58.823417 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" path="/var/lib/kubelet/pods/d3a4cda6-f9c8-441c-ad18-c335c59fb134/volumes" Oct 31 03:58:59.804775 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:59.804742 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6"] Oct 31 03:58:59.804994 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:58:59.804966 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" podUID="762ead81-eb44-4059-a19e-d6704ea7a111" containerName="maas-api" containerID="cri-o://256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0" gracePeriod=30 Oct 31 03:59:00.088221 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.088190 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:59:00.171463 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.171431 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762ead81-eb44-4059-a19e-d6704ea7a111-kube-api-access-9xbpc\" (UniqueName: \"kubernetes.io/projected/762ead81-eb44-4059-a19e-d6704ea7a111-kube-api-access-9xbpc\") pod \"762ead81-eb44-4059-a19e-d6704ea7a111\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " Oct 31 03:59:00.171657 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.171494 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls\") pod \"762ead81-eb44-4059-a19e-d6704ea7a111\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " Oct 31 03:59:00.171657 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.171552 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-metrics-certs\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-metrics-certs\") pod \"762ead81-eb44-4059-a19e-d6704ea7a111\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " Oct 31 03:59:00.171788 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.171682 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762ead81-eb44-4059-a19e-d6704ea7a111-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762ead81-eb44-4059-a19e-d6704ea7a111-openshift-service-ca\") pod \"762ead81-eb44-4059-a19e-d6704ea7a111\" (UID: \"762ead81-eb44-4059-a19e-d6704ea7a111\") " Oct 31 03:59:00.172493 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.172462 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762ead81-eb44-4059-a19e-d6704ea7a111-openshift-service-ca" pod "762ead81-eb44-4059-a19e-d6704ea7a111" (UID: "762ead81-eb44-4059-a19e-d6704ea7a111"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:00.173778 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.173752 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-metrics-certs" pod "762ead81-eb44-4059-a19e-d6704ea7a111" (UID: "762ead81-eb44-4059-a19e-d6704ea7a111"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:00.174258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.174200 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762ead81-eb44-4059-a19e-d6704ea7a111-kube-api-access-9xbpc" pod "762ead81-eb44-4059-a19e-d6704ea7a111" (UID: "762ead81-eb44-4059-a19e-d6704ea7a111"). InnerVolumeSpecName "kube-api-access-9xbpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:00.174258 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.174216 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls" pod "762ead81-eb44-4059-a19e-d6704ea7a111" (UID: "762ead81-eb44-4059-a19e-d6704ea7a111"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:00.273521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.273490 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:00.273521 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.273519 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762ead81-eb44-4059-a19e-d6704ea7a111-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:00.273714 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.273529 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762ead81-eb44-4059-a19e-d6704ea7a111-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:00.273714 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.273538 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbpc\" (UniqueName: \"kubernetes.io/projected/762ead81-eb44-4059-a19e-d6704ea7a111-kube-api-access-9xbpc\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:00.845491 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.845458 4934 generic.go:356] "Generic (PLEG): container finished" podID="762ead81-eb44-4059-a19e-d6704ea7a111" containerID="256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0" exitCode=0 Oct 31 03:59:00.845648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.845511 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" Oct 31 03:59:00.845648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.845547 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" event={"ID":"762ead81-eb44-4059-a19e-d6704ea7a111","Type":"ContainerDied","Data":"256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0"} Oct 31 03:59:00.845648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.845604 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6" event={"ID":"762ead81-eb44-4059-a19e-d6704ea7a111","Type":"ContainerDied","Data":"666273ccce31ba59b8d71e36c4e1a1ded5c71ddb5d51b96b64490d500284c314"} Oct 31 03:59:00.845648 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.845632 4934 scope.go:122] "RemoveContainer" containerID="256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0" Oct 31 03:59:00.855182 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.855164 4934 scope.go:122] "RemoveContainer" containerID="256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0" Oct 31 03:59:00.855519 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:00.855497 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0\": container with ID starting with 256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0 not found: ID does not exist" containerID="256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0" Oct 31 03:59:00.855616 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:00.855525 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0"} err="failed to get container status \"256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0\": rpc error: code = NotFound desc = could not find container \"256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0\": container with ID starting with 256bec681f93a3960bdb14c6cda27e4c0629fa8be4f2bf5e039edde286f67ab0 not found: ID does not exist" Oct 31 03:59:01.013342 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:01.013307 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6"] Oct 31 03:59:01.062826 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:01.062797 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05a9ff-7fb77f9d7-626l6"] Oct 31 03:59:02.822413 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:02.822384 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762ead81-eb44-4059-a19e-d6704ea7a111" path="/var/lib/kubelet/pods/762ead81-eb44-4059-a19e-d6704ea7a111/volumes" Oct 31 03:59:03.550771 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.550714 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" containerID="cri-o://c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8" gracePeriod=45 Oct 31 03:59:03.554818 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.554784 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" containerID="cri-o://5d6eaf2bd878d9f5364b1c74815a5bb13d0f617965afefc2455fe7ab4d7d3064" gracePeriod=45 Oct 31 03:59:03.574506 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.574459 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" containerID="cri-o://b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a" gracePeriod=45 Oct 31 03:59:03.731155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.731115 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": dial tcp 10.132.0.52:8000: connect: connection refused" Oct 31 03:59:03.854948 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.854924 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:59:03.858287 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.858161 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:59:03.865314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.865260 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" event={"ID":"6edb95a7-fa74-487f-ab8a-7658305c77ee","Type":"ContainerDied","Data":"c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8"} Oct 31 03:59:03.865314 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.865302 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" Oct 31 03:59:03.865444 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.865303 4934 scope.go:122] "RemoveContainer" containerID="c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8" Oct 31 03:59:03.865480 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.865260 4934 generic.go:356] "Generic (PLEG): container finished" podID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerID="c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8" exitCode=0 Oct 31 03:59:03.865611 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.865592 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b" event={"ID":"6edb95a7-fa74-487f-ab8a-7658305c77ee","Type":"ContainerDied","Data":"dddb8877c18c8fff38629f3c075b30d38324c00f7add0e641691202bac73ebb4"} Oct 31 03:59:03.867478 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.867432 4934 generic.go:356] "Generic (PLEG): container finished" podID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerID="b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a" exitCode=0 Oct 31 03:59:03.867687 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.867516 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" event={"ID":"17accd80-6cc2-4d3d-8e29-1db30b44852d","Type":"ContainerDied","Data":"b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a"} Oct 31 03:59:03.867687 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.867539 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" event={"ID":"17accd80-6cc2-4d3d-8e29-1db30b44852d","Type":"ContainerDied","Data":"2b5c6966868f0ee6e8ce44a6db9dfa108500ad0cda9edc183506190fb8a55b05"} Oct 31 03:59:03.867687 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.867584 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g" Oct 31 03:59:03.869599 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.869562 4934 generic.go:356] "Generic (PLEG): container finished" podID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerID="5d6eaf2bd878d9f5364b1c74815a5bb13d0f617965afefc2455fe7ab4d7d3064" exitCode=0 Oct 31 03:59:03.869698 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.869637 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" event={"ID":"03f4a9ca-e8db-4587-9a1a-08471c508625","Type":"ContainerDied","Data":"5d6eaf2bd878d9f5364b1c74815a5bb13d0f617965afefc2455fe7ab4d7d3064"} Oct 31 03:59:03.880380 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.880357 4934 scope.go:122] "RemoveContainer" containerID="c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8" Oct 31 03:59:03.880720 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:03.880694 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8\": container with ID starting with c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8 not found: ID does not exist" containerID="c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8" Oct 31 03:59:03.880802 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.880734 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8"} err="failed to get container status \"c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8\": rpc error: code = NotFound desc = could not find container \"c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8\": container with ID starting with c61b71e7c2e32cdf1be97f3dbff4afb1e3bd68ea1d6d9d96e9749679df4afaf8 not found: ID does not exist" Oct 31 03:59:03.880802 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.880754 4934 scope.go:122] "RemoveContainer" containerID="b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a" Oct 31 03:59:03.891630 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.891594 4934 scope.go:122] "RemoveContainer" containerID="b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a" Oct 31 03:59:03.892024 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:03.891943 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a\": container with ID starting with b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a not found: ID does not exist" containerID="b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a" Oct 31 03:59:03.892024 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.891971 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a"} err="failed to get container status \"b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a\": rpc error: code = NotFound desc = could not find container \"b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a\": container with ID starting with b784ada2bb78bcb4c6c0be983158e3b6c9b5742935fd0f355c0c950a82d85e9a not found: ID does not exist" Oct 31 03:59:03.900581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:03.900564 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:59:04.019983 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.019949 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-tmp-dir\") pod \"6edb95a7-fa74-487f-ab8a-7658305c77ee\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " Oct 31 03:59:04.020149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020006 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17accd80-6cc2-4d3d-8e29-1db30b44852d-kube-api-access-2vzdb\" (UniqueName: \"kubernetes.io/projected/17accd80-6cc2-4d3d-8e29-1db30b44852d-kube-api-access-2vzdb\") pod \"17accd80-6cc2-4d3d-8e29-1db30b44852d\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " Oct 31 03:59:04.020149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020025 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-tmp-dir\") pod \"03f4a9ca-e8db-4587-9a1a-08471c508625\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " Oct 31 03:59:04.020149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020041 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-dshm\") pod \"6edb95a7-fa74-487f-ab8a-7658305c77ee\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " Oct 31 03:59:04.020149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020056 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-home\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-home\") pod \"03f4a9ca-e8db-4587-9a1a-08471c508625\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " Oct 31 03:59:04.020149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020093 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-model-cache\") pod \"6edb95a7-fa74-487f-ab8a-7658305c77ee\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " Oct 31 03:59:04.020149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020117 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-tmp-dir\") pod \"17accd80-6cc2-4d3d-8e29-1db30b44852d\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " Oct 31 03:59:04.020149 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020142 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-dshm\") pod \"03f4a9ca-e8db-4587-9a1a-08471c508625\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020163 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-dshm\") pod \"17accd80-6cc2-4d3d-8e29-1db30b44852d\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020197 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-model-cache\") pod \"17accd80-6cc2-4d3d-8e29-1db30b44852d\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020216 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-home\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-home\") pod \"17accd80-6cc2-4d3d-8e29-1db30b44852d\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020263 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-model-cache\") pod \"03f4a9ca-e8db-4587-9a1a-08471c508625\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020292 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03f4a9ca-e8db-4587-9a1a-08471c508625-tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4a9ca-e8db-4587-9a1a-08471c508625-tls-certs\") pod \"03f4a9ca-e8db-4587-9a1a-08471c508625\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020332 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6edb95a7-fa74-487f-ab8a-7658305c77ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/6edb95a7-fa74-487f-ab8a-7658305c77ee-tls-certs\") pod \"6edb95a7-fa74-487f-ab8a-7658305c77ee\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020373 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-home\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-home\") pod \"6edb95a7-fa74-487f-ab8a-7658305c77ee\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020403 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edb95a7-fa74-487f-ab8a-7658305c77ee-kube-api-access-l2ctk\" (UniqueName: \"kubernetes.io/projected/6edb95a7-fa74-487f-ab8a-7658305c77ee-kube-api-access-l2ctk\") pod \"6edb95a7-fa74-487f-ab8a-7658305c77ee\" (UID: \"6edb95a7-fa74-487f-ab8a-7658305c77ee\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020425 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17accd80-6cc2-4d3d-8e29-1db30b44852d-tls-certs\" (UniqueName: \"kubernetes.io/secret/17accd80-6cc2-4d3d-8e29-1db30b44852d-tls-certs\") pod \"17accd80-6cc2-4d3d-8e29-1db30b44852d\" (UID: \"17accd80-6cc2-4d3d-8e29-1db30b44852d\") " Oct 31 03:59:04.020567 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.020440 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f4a9ca-e8db-4587-9a1a-08471c508625-kube-api-access-cl59d\" (UniqueName: \"kubernetes.io/projected/03f4a9ca-e8db-4587-9a1a-08471c508625-kube-api-access-cl59d\") pod \"03f4a9ca-e8db-4587-9a1a-08471c508625\" (UID: \"03f4a9ca-e8db-4587-9a1a-08471c508625\") " Oct 31 03:59:04.023154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.022400 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-tmp-dir" pod "6edb95a7-fa74-487f-ab8a-7658305c77ee" (UID: "6edb95a7-fa74-487f-ab8a-7658305c77ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.023154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.022610 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-model-cache" pod "03f4a9ca-e8db-4587-9a1a-08471c508625" (UID: "03f4a9ca-e8db-4587-9a1a-08471c508625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.023154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.022714 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-model-cache" pod "6edb95a7-fa74-487f-ab8a-7658305c77ee" (UID: "6edb95a7-fa74-487f-ab8a-7658305c77ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.023154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.022887 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-home" pod "03f4a9ca-e8db-4587-9a1a-08471c508625" (UID: "03f4a9ca-e8db-4587-9a1a-08471c508625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.023154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.023077 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-tmp-dir" pod "03f4a9ca-e8db-4587-9a1a-08471c508625" (UID: "03f4a9ca-e8db-4587-9a1a-08471c508625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.023154 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.023116 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-home" pod "6edb95a7-fa74-487f-ab8a-7658305c77ee" (UID: "6edb95a7-fa74-487f-ab8a-7658305c77ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.024683 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.024323 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-model-cache" pod "17accd80-6cc2-4d3d-8e29-1db30b44852d" (UID: "17accd80-6cc2-4d3d-8e29-1db30b44852d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.024683 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.024345 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-tmp-dir" pod "17accd80-6cc2-4d3d-8e29-1db30b44852d" (UID: "17accd80-6cc2-4d3d-8e29-1db30b44852d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.024683 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.024630 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-home" pod "17accd80-6cc2-4d3d-8e29-1db30b44852d" (UID: "17accd80-6cc2-4d3d-8e29-1db30b44852d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.026045 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.026008 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f4a9ca-e8db-4587-9a1a-08471c508625-tls-certs" pod "03f4a9ca-e8db-4587-9a1a-08471c508625" (UID: "03f4a9ca-e8db-4587-9a1a-08471c508625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:04.027557 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.027526 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edb95a7-fa74-487f-ab8a-7658305c77ee-tls-certs" pod "6edb95a7-fa74-487f-ab8a-7658305c77ee" (UID: "6edb95a7-fa74-487f-ab8a-7658305c77ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:04.027636 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.027591 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f4a9ca-e8db-4587-9a1a-08471c508625-kube-api-access-cl59d" pod "03f4a9ca-e8db-4587-9a1a-08471c508625" (UID: "03f4a9ca-e8db-4587-9a1a-08471c508625"). InnerVolumeSpecName "kube-api-access-cl59d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:04.028710 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.028678 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17accd80-6cc2-4d3d-8e29-1db30b44852d-kube-api-access-2vzdb" pod "17accd80-6cc2-4d3d-8e29-1db30b44852d" (UID: "17accd80-6cc2-4d3d-8e29-1db30b44852d"). InnerVolumeSpecName "kube-api-access-2vzdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:04.028710 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.028702 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-dshm" pod "03f4a9ca-e8db-4587-9a1a-08471c508625" (UID: "03f4a9ca-e8db-4587-9a1a-08471c508625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.028864 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.028754 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edb95a7-fa74-487f-ab8a-7658305c77ee-kube-api-access-l2ctk" pod "6edb95a7-fa74-487f-ab8a-7658305c77ee" (UID: "6edb95a7-fa74-487f-ab8a-7658305c77ee"). InnerVolumeSpecName "kube-api-access-l2ctk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:04.028927 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.028882 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-dshm" pod "6edb95a7-fa74-487f-ab8a-7658305c77ee" (UID: "6edb95a7-fa74-487f-ab8a-7658305c77ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.029133 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.029106 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17accd80-6cc2-4d3d-8e29-1db30b44852d-tls-certs" pod "17accd80-6cc2-4d3d-8e29-1db30b44852d" (UID: "17accd80-6cc2-4d3d-8e29-1db30b44852d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:04.029635 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.029611 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-dshm" pod "17accd80-6cc2-4d3d-8e29-1db30b44852d" (UID: "17accd80-6cc2-4d3d-8e29-1db30b44852d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.121885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121808 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.121885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121835 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4a9ca-e8db-4587-9a1a-08471c508625-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.121885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121849 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edb95a7-fa74-487f-ab8a-7658305c77ee-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.121885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121862 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.121885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121875 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ctk\" (UniqueName: \"kubernetes.io/projected/6edb95a7-fa74-487f-ab8a-7658305c77ee-kube-api-access-l2ctk\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.121885 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121888 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17accd80-6cc2-4d3d-8e29-1db30b44852d-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121902 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl59d\" (UniqueName: \"kubernetes.io/projected/03f4a9ca-e8db-4587-9a1a-08471c508625-kube-api-access-cl59d\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121917 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121931 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vzdb\" (UniqueName: \"kubernetes.io/projected/17accd80-6cc2-4d3d-8e29-1db30b44852d-kube-api-access-2vzdb\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121946 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121956 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121970 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121982 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edb95a7-fa74-487f-ab8a-7658305c77ee-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.121996 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.122009 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4a9ca-e8db-4587-9a1a-08471c508625-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.122023 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.122035 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.122305 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.122046 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17accd80-6cc2-4d3d-8e29-1db30b44852d-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.374369 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.374294 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b"] Oct 31 03:59:04.379035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.378997 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-2-w3-b98893-kserve-7c8644cbbd-dzr9b"] Oct 31 03:59:04.488093 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.488069 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g"] Oct 31 03:59:04.497051 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.497025 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/simulator-w3-b98893-kserve-57fc5ff9cc-2dc8g"] Oct 31 03:59:04.824751 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.824722 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" path="/var/lib/kubelet/pods/17accd80-6cc2-4d3d-8e29-1db30b44852d/volumes" Oct 31 03:59:04.825167 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.825152 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" path="/var/lib/kubelet/pods/6edb95a7-fa74-487f-ab8a-7658305c77ee/volumes" Oct 31 03:59:04.875830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.875808 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" Oct 31 03:59:04.875830 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.875820 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc" event={"ID":"03f4a9ca-e8db-4587-9a1a-08471c508625","Type":"ContainerDied","Data":"28d6ed7827a1a58cce11e4c9680b065bc92d7a4b7c5f4e7f8111390a2e3403e8"} Oct 31 03:59:04.876307 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:04.875859 4934 scope.go:122] "RemoveContainer" containerID="5d6eaf2bd878d9f5364b1c74815a5bb13d0f617965afefc2455fe7ab4d7d3064" Oct 31 03:59:05.040565 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:05.040539 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc"] Oct 31 03:59:05.044708 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:05.044689 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/distinct-w3-b98893-kserve-75fc645c4c-r4htc"] Oct 31 03:59:06.821072 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:06.820992 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" path="/var/lib/kubelet/pods/03f4a9ca-e8db-4587-9a1a-08471c508625/volumes" Oct 31 03:59:09.028690 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:09.028650 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8"] Oct 31 03:59:09.029155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:09.028974 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" containerID="cri-o://0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7" gracePeriod=30 Oct 31 03:59:12.011144 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:12.011107 4934 patch_prober.go:29] interesting pod/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:59:12.011577 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:12.011166 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:59:14.509261 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.508811 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh"] Oct 31 03:59:14.551076 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.551054 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:59:14.737197 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737164 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-podinfo\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737374 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737214 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-kube-api-access-66kdz\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-kube-api-access-66kdz\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737374 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737266 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-data\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737374 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737300 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-certs\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737496 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737465 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-socket\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737575 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737554 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-envoy\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737555 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-data" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:14.737646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737592 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istiod-ca-cert\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737646 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737585 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-certs" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:14.737800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737670 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-token\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-token\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737713 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-socket" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:14.737800 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737756 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-credential-socket\") pod \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\" (UID: \"365b8a0d-2476-4e6f-95fa-ddcbad399dca\") " Oct 31 03:59:14.737965 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737878 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-credential-socket" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:14.737965 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.737886 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istiod-ca-cert" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:14.738177 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.738138 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.738443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.738182 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.738443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.738193 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.738443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.738202 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.738443 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.738211 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.739677 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.739649 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-podinfo" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:59:14.739920 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.739905 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-token" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:14.740186 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.740159 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-envoy" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:14.740309 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.740199 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-kube-api-access-66kdz" pod "365b8a0d-2476-4e6f-95fa-ddcbad399dca" (UID: "365b8a0d-2476-4e6f-95fa-ddcbad399dca"). InnerVolumeSpecName "kube-api-access-66kdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:14.839453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.839351 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.839453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.839385 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.839453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.839400 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66kdz\" (UniqueName: \"kubernetes.io/projected/365b8a0d-2476-4e6f-95fa-ddcbad399dca-kube-api-access-66kdz\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.839453 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.839415 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365b8a0d-2476-4e6f-95fa-ddcbad399dca-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.937599 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.937565 4934 generic.go:356] "Generic (PLEG): container finished" podID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerID="0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7" exitCode=0 Oct 31 03:59:14.937754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.937625 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" Oct 31 03:59:14.937754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.937662 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" event={"ID":"365b8a0d-2476-4e6f-95fa-ddcbad399dca","Type":"ContainerDied","Data":"0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7"} Oct 31 03:59:14.937754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.937719 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8" event={"ID":"365b8a0d-2476-4e6f-95fa-ddcbad399dca","Type":"ContainerDied","Data":"73b4f18938454dc0deafa7f77167f9474256fa4689a2cc644339ec46aa796d91"} Oct 31 03:59:14.937754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.937746 4934 scope.go:122] "RemoveContainer" containerID="0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7" Oct 31 03:59:14.946968 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.946951 4934 scope.go:122] "RemoveContainer" containerID="58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68" Oct 31 03:59:14.956290 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.956010 4934 scope.go:122] "RemoveContainer" containerID="0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7" Oct 31 03:59:14.956802 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:14.956528 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7\": container with ID starting with 0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7 not found: ID does not exist" containerID="0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7" Oct 31 03:59:14.956802 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.956562 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7"} err="failed to get container status \"0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7\": rpc error: code = NotFound desc = could not find container \"0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7\": container with ID starting with 0b8997dae43b5f010fe042db1442ad88032a94f2d39a50b01dc4e9e9cc72d7f7 not found: ID does not exist" Oct 31 03:59:14.956802 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.956587 4934 scope.go:122] "RemoveContainer" containerID="58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68" Oct 31 03:59:14.957307 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:14.957265 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68\": container with ID starting with 58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68 not found: ID does not exist" containerID="58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68" Oct 31 03:59:14.957980 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:14.957938 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68"} err="failed to get container status \"58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68\": rpc error: code = NotFound desc = could not find container \"58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68\": container with ID starting with 58c2a5113ef17643fc954545abdc930d4e09ce4eb266039c796daa508a5dad68 not found: ID does not exist" Oct 31 03:59:15.119730 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:15.119647 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8"] Oct 31 03:59:15.130284 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:15.129524 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-05a9ff-openshift-default-7999d6bb69-b6gx8"] Oct 31 03:59:16.821371 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:16.821339 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" path="/var/lib/kubelet/pods/365b8a0d-2476-4e6f-95fa-ddcbad399dca/volumes" Oct 31 03:59:19.164561 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:19.164522 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:59:27.672770 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:27.672742 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:59:27.733534 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:27.733504 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 03:59:29.580841 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.580779 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" containerID="cri-o://9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a" gracePeriod=45 Oct 31 03:59:29.836470 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.836411 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:59:29.897119 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897082 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-model-cache\") pod \"61fa25de-5c01-4e68-b69d-91699978b69f\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " Oct 31 03:59:29.897322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897135 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-dshm\") pod \"61fa25de-5c01-4e68-b69d-91699978b69f\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " Oct 31 03:59:29.897322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897191 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-tmp-dir\") pod \"61fa25de-5c01-4e68-b69d-91699978b69f\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " Oct 31 03:59:29.897322 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897240 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61fa25de-5c01-4e68-b69d-91699978b69f-tls-certs\" (UniqueName: \"kubernetes.io/secret/61fa25de-5c01-4e68-b69d-91699978b69f-tls-certs\") pod \"61fa25de-5c01-4e68-b69d-91699978b69f\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " Oct 31 03:59:29.897497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897342 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fa25de-5c01-4e68-b69d-91699978b69f-kube-api-access-lfw2s\" (UniqueName: \"kubernetes.io/projected/61fa25de-5c01-4e68-b69d-91699978b69f-kube-api-access-lfw2s\") pod \"61fa25de-5c01-4e68-b69d-91699978b69f\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " Oct 31 03:59:29.897497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897374 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-model-cache" pod "61fa25de-5c01-4e68-b69d-91699978b69f" (UID: "61fa25de-5c01-4e68-b69d-91699978b69f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:29.897497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897409 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-home\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-home\") pod \"61fa25de-5c01-4e68-b69d-91699978b69f\" (UID: \"61fa25de-5c01-4e68-b69d-91699978b69f\") " Oct 31 03:59:29.897497 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897475 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-tmp-dir" pod "61fa25de-5c01-4e68-b69d-91699978b69f" (UID: "61fa25de-5c01-4e68-b69d-91699978b69f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:29.897754 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897734 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-home" pod "61fa25de-5c01-4e68-b69d-91699978b69f" (UID: "61fa25de-5c01-4e68-b69d-91699978b69f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:29.897810 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897795 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:29.897863 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.897838 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:29.899651 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.899618 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-dshm" pod "61fa25de-5c01-4e68-b69d-91699978b69f" (UID: "61fa25de-5c01-4e68-b69d-91699978b69f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:29.899935 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.899913 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fa25de-5c01-4e68-b69d-91699978b69f-tls-certs" pod "61fa25de-5c01-4e68-b69d-91699978b69f" (UID: "61fa25de-5c01-4e68-b69d-91699978b69f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:29.900121 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.900105 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fa25de-5c01-4e68-b69d-91699978b69f-kube-api-access-lfw2s" pod "61fa25de-5c01-4e68-b69d-91699978b69f" (UID: "61fa25de-5c01-4e68-b69d-91699978b69f"). InnerVolumeSpecName "kube-api-access-lfw2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:29.998817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.998784 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:29.998817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.998829 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fa25de-5c01-4e68-b69d-91699978b69f-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:29.999026 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.998847 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfw2s\" (UniqueName: \"kubernetes.io/projected/61fa25de-5c01-4e68-b69d-91699978b69f-kube-api-access-lfw2s\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:29.999026 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:29.998860 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fa25de-5c01-4e68-b69d-91699978b69f-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:30.023668 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.023636 4934 generic.go:356] "Generic (PLEG): container finished" podID="61fa25de-5c01-4e68-b69d-91699978b69f" containerID="9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a" exitCode=0 Oct 31 03:59:30.023848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.023707 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" Oct 31 03:59:30.023848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.023710 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" event={"ID":"61fa25de-5c01-4e68-b69d-91699978b69f","Type":"ContainerDied","Data":"9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a"} Oct 31 03:59:30.023848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.023769 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh" event={"ID":"61fa25de-5c01-4e68-b69d-91699978b69f","Type":"ContainerDied","Data":"d8777cf51cdee958871b2c064ba3d65893775241488e99c8fa2e1b4ce0a3a5bf"} Oct 31 03:59:30.023848 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.023793 4934 scope.go:122] "RemoveContainer" containerID="9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a" Oct 31 03:59:30.033072 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.033056 4934 scope.go:122] "RemoveContainer" containerID="9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a" Oct 31 03:59:30.033315 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:30.033298 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a\": container with ID starting with 9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a not found: ID does not exist" containerID="9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a" Oct 31 03:59:30.033394 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.033323 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a"} err="failed to get container status \"9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a\": rpc error: code = NotFound desc = could not find container \"9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a\": container with ID starting with 9d38980bbea503901655f8cb5135d20fae97d909b9def06878321d687c65f03a not found: ID does not exist" Oct 31 03:59:30.187678 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.187596 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh"] Oct 31 03:59:30.192539 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.192509 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-2-w0-909365-kserve-674b8ccd86-qpdgh"] Oct 31 03:59:30.821341 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:30.821315 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" path="/var/lib/kubelet/pods/61fa25de-5c01-4e68-b69d-91699978b69f/volumes" Oct 31 03:59:31.352901 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:31.352866 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm"] Oct 31 03:59:31.353132 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:31.353102 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" containerID="cri-o://2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6" gracePeriod=30 Oct 31 03:59:36.616351 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.616325 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:59:36.692817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.692743 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-certs\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.692817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.692787 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-istio-token\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-istio-token\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.692858 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37c01246-a320-4e53-97f9-f26a2591547c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c01246-a320-4e53-97f9-f26a2591547c-istio-podinfo\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.692887 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-envoy\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.692954 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-credential-socket\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.692981 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c01246-a320-4e53-97f9-f26a2591547c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c01246-a320-4e53-97f9-f26a2591547c-istiod-ca-cert\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693012 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.692995 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-certs" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:36.693292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693021 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-data\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693082 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-socket\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693115 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-kube-api-access-6dr5k\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-kube-api-access-6dr5k\") pod \"37c01246-a320-4e53-97f9-f26a2591547c\" (UID: \"37c01246-a320-4e53-97f9-f26a2591547c\") " Oct 31 03:59:36.693292 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693218 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-credential-socket" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:36.693495 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693365 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c01246-a320-4e53-97f9-f26a2591547c-istiod-ca-cert" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:36.693495 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693428 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-socket" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:36.693495 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693478 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-data" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:36.693763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693730 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.693763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693748 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.693763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693760 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c01246-a320-4e53-97f9-f26a2591547c-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.693763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693774 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.693763 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.693786 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.695504 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.695483 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-envoy" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:36.695603 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.695512 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-kube-api-access-6dr5k" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "kube-api-access-6dr5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:36.695653 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.695635 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-istio-token" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:36.695689 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.695624 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37c01246-a320-4e53-97f9-f26a2591547c-istio-podinfo" pod "37c01246-a320-4e53-97f9-f26a2591547c" (UID: "37c01246-a320-4e53-97f9-f26a2591547c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:59:36.795082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.795053 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c01246-a320-4e53-97f9-f26a2591547c-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.795082 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.795081 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c01246-a320-4e53-97f9-f26a2591547c-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.795291 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.795090 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dr5k\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-kube-api-access-6dr5k\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.795291 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.795101 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c01246-a320-4e53-97f9-f26a2591547c-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.873893 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.873863 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6"] Oct 31 03:59:36.883919 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:36.883893 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf"] Oct 31 03:59:37.065333 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.065303 4934 generic.go:356] "Generic (PLEG): container finished" podID="37c01246-a320-4e53-97f9-f26a2591547c" containerID="2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6" exitCode=0 Oct 31 03:59:37.065333 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.065343 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" event={"ID":"37c01246-a320-4e53-97f9-f26a2591547c","Type":"ContainerDied","Data":"2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6"} Oct 31 03:59:37.065577 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.065357 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" Oct 31 03:59:37.065577 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.065385 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm" event={"ID":"37c01246-a320-4e53-97f9-f26a2591547c","Type":"ContainerDied","Data":"71dcd9f3878e98cc3eeece2c05f4168e3b571dbdb8aa39feabc3a49a38fde6ba"} Oct 31 03:59:37.065577 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.065409 4934 scope.go:122] "RemoveContainer" containerID="2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6" Oct 31 03:59:37.074523 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.074500 4934 scope.go:122] "RemoveContainer" containerID="37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a" Oct 31 03:59:37.082531 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.082512 4934 scope.go:122] "RemoveContainer" containerID="2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6" Oct 31 03:59:37.082804 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:37.082778 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6\": container with ID starting with 2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6 not found: ID does not exist" containerID="2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6" Oct 31 03:59:37.082866 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.082810 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6"} err="failed to get container status \"2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6\": rpc error: code = NotFound desc = could not find container \"2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6\": container with ID starting with 2c11f7a092aa8653dc5f8c4dc001b67e3476087b586dbd51e5dfac3c3ee5ede6 not found: ID does not exist" Oct 31 03:59:37.082866 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.082830 4934 scope.go:122] "RemoveContainer" containerID="37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a" Oct 31 03:59:37.083087 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:37.083069 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a\": container with ID starting with 37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a not found: ID does not exist" containerID="37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a" Oct 31 03:59:37.083138 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.083098 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a"} err="failed to get container status \"37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a\": rpc error: code = NotFound desc = could not find container \"37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a\": container with ID starting with 37ade6639178a0730c406ec62c22483fb06835074e2a58d15ea47e867afe637a not found: ID does not exist" Oct 31 03:59:37.220075 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.220036 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm"] Oct 31 03:59:37.224872 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:37.224845 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b16de9-openshift-default-6dc96f7cf7-qwxcm"] Oct 31 03:59:38.821787 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:38.821759 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c01246-a320-4e53-97f9-f26a2591547c" path="/var/lib/kubelet/pods/37c01246-a320-4e53-97f9-f26a2591547c/volumes" Oct 31 03:59:49.164300 ip-10-0-138-238 kubenswrapper[4934]: E1031 03:59:49.164267 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 03:59:50.908610 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:50.908574 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x"] Oct 31 03:59:50.909073 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:50.908765 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" podUID="c44742f8-e136-48da-95c6-8cd88b73e76f" containerName="maas-api" containerID="cri-o://0eef1d0251578d33ed1ef9d39d0d13e65ade46192efd9b217b5bdcf3249e89e9" gracePeriod=30 Oct 31 03:59:51.155253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.154723 4934 generic.go:356] "Generic (PLEG): container finished" podID="c44742f8-e136-48da-95c6-8cd88b73e76f" containerID="0eef1d0251578d33ed1ef9d39d0d13e65ade46192efd9b217b5bdcf3249e89e9" exitCode=0 Oct 31 03:59:51.155253 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.154872 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" event={"ID":"c44742f8-e136-48da-95c6-8cd88b73e76f","Type":"ContainerDied","Data":"0eef1d0251578d33ed1ef9d39d0d13e65ade46192efd9b217b5bdcf3249e89e9"} Oct 31 03:59:51.210393 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.210361 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:59:51.244776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.244748 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls\") pod \"c44742f8-e136-48da-95c6-8cd88b73e76f\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " Oct 31 03:59:51.244921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.244863 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44742f8-e136-48da-95c6-8cd88b73e76f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44742f8-e136-48da-95c6-8cd88b73e76f-openshift-service-ca\") pod \"c44742f8-e136-48da-95c6-8cd88b73e76f\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " Oct 31 03:59:51.244921 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.244908 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44742f8-e136-48da-95c6-8cd88b73e76f-kube-api-access-8rjdh\" (UniqueName: \"kubernetes.io/projected/c44742f8-e136-48da-95c6-8cd88b73e76f-kube-api-access-8rjdh\") pod \"c44742f8-e136-48da-95c6-8cd88b73e76f\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " Oct 31 03:59:51.245051 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.245011 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-metrics-certs\") pod \"c44742f8-e136-48da-95c6-8cd88b73e76f\" (UID: \"c44742f8-e136-48da-95c6-8cd88b73e76f\") " Oct 31 03:59:51.245498 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.245435 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44742f8-e136-48da-95c6-8cd88b73e76f-openshift-service-ca" pod "c44742f8-e136-48da-95c6-8cd88b73e76f" (UID: "c44742f8-e136-48da-95c6-8cd88b73e76f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:51.249855 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.249822 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls" pod "c44742f8-e136-48da-95c6-8cd88b73e76f" (UID: "c44742f8-e136-48da-95c6-8cd88b73e76f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:51.250001 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.249981 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-metrics-certs" pod "c44742f8-e136-48da-95c6-8cd88b73e76f" (UID: "c44742f8-e136-48da-95c6-8cd88b73e76f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:51.250447 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.250407 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44742f8-e136-48da-95c6-8cd88b73e76f-kube-api-access-8rjdh" pod "c44742f8-e136-48da-95c6-8cd88b73e76f" (UID: "c44742f8-e136-48da-95c6-8cd88b73e76f"). InnerVolumeSpecName "kube-api-access-8rjdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:51.347937 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.347897 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44742f8-e136-48da-95c6-8cd88b73e76f-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:51.347937 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.347937 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjdh\" (UniqueName: \"kubernetes.io/projected/c44742f8-e136-48da-95c6-8cd88b73e76f-kube-api-access-8rjdh\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:51.353581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.347956 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:51.353581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.347969 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44742f8-e136-48da-95c6-8cd88b73e76f-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:51.895155 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.894535 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" containerID="cri-o://a92f1ef95f3c0243d85fa8c9612a111d07b96b2fdfebb43802ed8fbeebaaba18" gracePeriod=45 Oct 31 03:59:51.904817 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:51.904671 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" containerID="cri-o://467e2c6bd7ea3fadc9b070f88b66ee11ca78a0422180b3da19a3cfea867af6d1" gracePeriod=45 Oct 31 03:59:52.068589 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.068354 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": dial tcp 10.132.0.45:8000: connect: connection refused" Oct 31 03:59:52.163279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.163157 4934 generic.go:356] "Generic (PLEG): container finished" podID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerID="467e2c6bd7ea3fadc9b070f88b66ee11ca78a0422180b3da19a3cfea867af6d1" exitCode=0 Oct 31 03:59:52.163279 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.163245 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" event={"ID":"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14","Type":"ContainerDied","Data":"467e2c6bd7ea3fadc9b070f88b66ee11ca78a0422180b3da19a3cfea867af6d1"} Oct 31 03:59:52.164996 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.164941 4934 generic.go:356] "Generic (PLEG): container finished" podID="b37a6959-4519-4d12-aa29-86651e36705b" containerID="a92f1ef95f3c0243d85fa8c9612a111d07b96b2fdfebb43802ed8fbeebaaba18" exitCode=0 Oct 31 03:59:52.165117 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.165034 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" event={"ID":"b37a6959-4519-4d12-aa29-86651e36705b","Type":"ContainerDied","Data":"a92f1ef95f3c0243d85fa8c9612a111d07b96b2fdfebb43802ed8fbeebaaba18"} Oct 31 03:59:52.169705 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.166485 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" event={"ID":"c44742f8-e136-48da-95c6-8cd88b73e76f","Type":"ContainerDied","Data":"e85ebb883f7eca8e2ab964914bbce5b16a3cd9191e1bf0dd0a755a4ff23a7854"} Oct 31 03:59:52.169705 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.166549 4934 scope.go:122] "RemoveContainer" containerID="0eef1d0251578d33ed1ef9d39d0d13e65ade46192efd9b217b5bdcf3249e89e9" Oct 31 03:59:52.169705 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.166746 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x" Oct 31 03:59:52.196172 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.196144 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:59:52.233974 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.233951 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:59:52.286165 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286110 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tls-certs\") pod \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " Oct 31 03:59:52.286372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286198 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tmp-dir\") pod \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " Oct 31 03:59:52.286372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286285 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37a6959-4519-4d12-aa29-86651e36705b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b37a6959-4519-4d12-aa29-86651e36705b-tls-certs\") pod \"b37a6959-4519-4d12-aa29-86651e36705b\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " Oct 31 03:59:52.286372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286336 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-model-cache\") pod \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " Oct 31 03:59:52.286372 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286368 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-dshm\") pod \"b37a6959-4519-4d12-aa29-86651e36705b\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " Oct 31 03:59:52.286613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286434 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-home\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-home\") pod \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " Oct 31 03:59:52.286613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286460 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-tmp-dir\") pod \"b37a6959-4519-4d12-aa29-86651e36705b\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " Oct 31 03:59:52.286613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286509 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-dshm\") pod \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " Oct 31 03:59:52.286613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286537 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-home\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-home\") pod \"b37a6959-4519-4d12-aa29-86651e36705b\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " Oct 31 03:59:52.286613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286567 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-kube-api-access-zqlcz\" (UniqueName: \"kubernetes.io/projected/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-kube-api-access-zqlcz\") pod \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\" (UID: \"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14\") " Oct 31 03:59:52.286613 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286454 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tmp-dir" pod "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" (UID: "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.286934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286615 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-model-cache\") pod \"b37a6959-4519-4d12-aa29-86651e36705b\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " Oct 31 03:59:52.286934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286618 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-model-cache" pod "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" (UID: "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.286934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286662 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37a6959-4519-4d12-aa29-86651e36705b-kube-api-access-pgmmq\" (UniqueName: \"kubernetes.io/projected/b37a6959-4519-4d12-aa29-86651e36705b-kube-api-access-pgmmq\") pod \"b37a6959-4519-4d12-aa29-86651e36705b\" (UID: \"b37a6959-4519-4d12-aa29-86651e36705b\") " Oct 31 03:59:52.286934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286830 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-tmp-dir" pod "b37a6959-4519-4d12-aa29-86651e36705b" (UID: "b37a6959-4519-4d12-aa29-86651e36705b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.286934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286838 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-home" pod "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" (UID: "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.286934 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.286897 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-home" pod "b37a6959-4519-4d12-aa29-86651e36705b" (UID: "b37a6959-4519-4d12-aa29-86651e36705b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.287454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.287093 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.287454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.287010 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-model-cache" pod "b37a6959-4519-4d12-aa29-86651e36705b" (UID: "b37a6959-4519-4d12-aa29-86651e36705b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.287454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.287137 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.287454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.287166 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.287454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.287184 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.287454 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.287202 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.289033 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.288999 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37a6959-4519-4d12-aa29-86651e36705b-tls-certs" pod "b37a6959-4519-4d12-aa29-86651e36705b" (UID: "b37a6959-4519-4d12-aa29-86651e36705b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:52.289346 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.289303 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-dshm" pod "b37a6959-4519-4d12-aa29-86651e36705b" (UID: "b37a6959-4519-4d12-aa29-86651e36705b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.289457 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.289394 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a6959-4519-4d12-aa29-86651e36705b-kube-api-access-pgmmq" pod "b37a6959-4519-4d12-aa29-86651e36705b" (UID: "b37a6959-4519-4d12-aa29-86651e36705b"). InnerVolumeSpecName "kube-api-access-pgmmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:52.289698 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.289681 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tls-certs" pod "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" (UID: "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:52.293666 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.293640 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-dshm" pod "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" (UID: "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.293776 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.293672 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-kube-api-access-zqlcz" pod "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" (UID: "4d07ca2b-173c-4293-bc8f-2f8e5bf95e14"). InnerVolumeSpecName "kube-api-access-zqlcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:52.334196 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.334167 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x"] Oct 31 03:59:52.383451 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.383426 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b98893-666759d68c-95m9x"] Oct 31 03:59:52.389659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.389629 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.389659 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.389661 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqlcz\" (UniqueName: \"kubernetes.io/projected/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-kube-api-access-zqlcz\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.389891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.389674 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.389891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.389690 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgmmq\" (UniqueName: \"kubernetes.io/projected/b37a6959-4519-4d12-aa29-86651e36705b-kube-api-access-pgmmq\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.389891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.389704 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.389891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.389716 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37a6959-4519-4d12-aa29-86651e36705b-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.389891 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.389728 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37a6959-4519-4d12-aa29-86651e36705b-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.820145 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:52.820108 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44742f8-e136-48da-95c6-8cd88b73e76f" path="/var/lib/kubelet/pods/c44742f8-e136-48da-95c6-8cd88b73e76f/volumes" Oct 31 03:59:53.173035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.172917 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" Oct 31 03:59:53.173035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.172957 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf" event={"ID":"4d07ca2b-173c-4293-bc8f-2f8e5bf95e14","Type":"ContainerDied","Data":"9ac8b830fcd6e97fe6192eeeab2f6820db1d2ebc881b8c2cbb5ea1fca9c7441d"} Oct 31 03:59:53.173035 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.173020 4934 scope.go:122] "RemoveContainer" containerID="467e2c6bd7ea3fadc9b070f88b66ee11ca78a0422180b3da19a3cfea867af6d1" Oct 31 03:59:53.175586 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.175534 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" event={"ID":"b37a6959-4519-4d12-aa29-86651e36705b","Type":"ContainerDied","Data":"630329d940b62a16261f74ae8b373ceefe635ab9cc46aae7cc198e81c0506404"} Oct 31 03:59:53.175833 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.175517 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6" Oct 31 03:59:53.183581 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.183558 4934 scope.go:122] "RemoveContainer" containerID="a92f1ef95f3c0243d85fa8c9612a111d07b96b2fdfebb43802ed8fbeebaaba18" Oct 31 03:59:53.336501 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.336471 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf"] Oct 31 03:59:53.369601 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.369570 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/simulator-w2-dec4a0-kserve-7c8885b858-db8bf"] Oct 31 03:59:53.450568 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.450495 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6"] Oct 31 03:59:53.455106 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:53.455075 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/premium-w2-dec4a0-kserve-7d884d7b5f-58ms6"] Oct 31 03:59:54.820676 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:54.820645 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" path="/var/lib/kubelet/pods/4d07ca2b-173c-4293-bc8f-2f8e5bf95e14/volumes" Oct 31 03:59:54.821068 ip-10-0-138-238 kubenswrapper[4934]: I1031 03:59:54.821008 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37a6959-4519-4d12-aa29-86651e36705b" path="/var/lib/kubelet/pods/b37a6959-4519-4d12-aa29-86651e36705b/volumes" Oct 31 04:00:00.688366 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:00.688333 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq"] Oct 31 04:00:00.688920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:00.688567 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" containerID="cri-o://8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe" gracePeriod=30 Oct 31 04:00:02.160430 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.160395 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc"] Oct 31 04:00:02.160838 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.160588 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" podUID="febda22b-0402-4a68-a578-4202936a45b7" containerName="maas-api" containerID="cri-o://508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41" gracePeriod=30 Oct 31 04:00:02.430946 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.430915 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 04:00:02.502518 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.502486 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls\") pod \"febda22b-0402-4a68-a578-4202936a45b7\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " Oct 31 04:00:02.502701 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.502553 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/febda22b-0402-4a68-a578-4202936a45b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febda22b-0402-4a68-a578-4202936a45b7-openshift-service-ca\") pod \"febda22b-0402-4a68-a578-4202936a45b7\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " Oct 31 04:00:02.502701 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.502638 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febda22b-0402-4a68-a578-4202936a45b7-kube-api-access-fmbl2\" (UniqueName: \"kubernetes.io/projected/febda22b-0402-4a68-a578-4202936a45b7-kube-api-access-fmbl2\") pod \"febda22b-0402-4a68-a578-4202936a45b7\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " Oct 31 04:00:02.502701 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.502671 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-metrics-certs\") pod \"febda22b-0402-4a68-a578-4202936a45b7\" (UID: \"febda22b-0402-4a68-a578-4202936a45b7\") " Oct 31 04:00:02.502915 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.502887 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febda22b-0402-4a68-a578-4202936a45b7-openshift-service-ca" pod "febda22b-0402-4a68-a578-4202936a45b7" (UID: "febda22b-0402-4a68-a578-4202936a45b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:02.504821 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.504794 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febda22b-0402-4a68-a578-4202936a45b7-kube-api-access-fmbl2" pod "febda22b-0402-4a68-a578-4202936a45b7" (UID: "febda22b-0402-4a68-a578-4202936a45b7"). InnerVolumeSpecName "kube-api-access-fmbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:02.504935 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.504825 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls" pod "febda22b-0402-4a68-a578-4202936a45b7" (UID: "febda22b-0402-4a68-a578-4202936a45b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:02.504935 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.504845 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-metrics-certs" pod "febda22b-0402-4a68-a578-4202936a45b7" (UID: "febda22b-0402-4a68-a578-4202936a45b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:02.604241 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.604196 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmbl2\" (UniqueName: \"kubernetes.io/projected/febda22b-0402-4a68-a578-4202936a45b7-kube-api-access-fmbl2\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:02.604241 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.604244 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:02.604443 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.604258 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febda22b-0402-4a68-a578-4202936a45b7-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:02.604443 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:02.604271 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febda22b-0402-4a68-a578-4202936a45b7-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:03.235276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.235166 4934 generic.go:356] "Generic (PLEG): container finished" podID="febda22b-0402-4a68-a578-4202936a45b7" containerID="508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41" exitCode=0 Oct 31 04:00:03.235276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.235216 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" Oct 31 04:00:03.235276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.235252 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" event={"ID":"febda22b-0402-4a68-a578-4202936a45b7","Type":"ContainerDied","Data":"508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41"} Oct 31 04:00:03.235789 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.235320 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc" event={"ID":"febda22b-0402-4a68-a578-4202936a45b7","Type":"ContainerDied","Data":"1c1b48d0c9a80c1d3b637ae1d65e2db667b08adfe030f34f1f50b38d8d52890d"} Oct 31 04:00:03.235789 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.235346 4934 scope.go:122] "RemoveContainer" containerID="508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41" Oct 31 04:00:03.244002 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.243973 4934 scope.go:122] "RemoveContainer" containerID="508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41" Oct 31 04:00:03.244289 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:00:03.244270 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41\": container with ID starting with 508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41 not found: ID does not exist" containerID="508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41" Oct 31 04:00:03.244359 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.244300 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41"} err="failed to get container status \"508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41\": rpc error: code = NotFound desc = could not find container \"508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41\": container with ID starting with 508e55f10129e978d17cb61844cef656c07ff52134acab8eb7625b1c0eb7cc41 not found: ID does not exist" Oct 31 04:00:03.400156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.400122 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc"] Oct 31 04:00:03.450127 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:03.450077 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-909365-55fc57869-ndjlc"] Oct 31 04:00:04.820481 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:04.820451 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febda22b-0402-4a68-a578-4202936a45b7" path="/var/lib/kubelet/pods/febda22b-0402-4a68-a578-4202936a45b7/volumes" Oct 31 04:00:05.941449 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:05.941418 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 04:00:06.037440 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037355 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-kube-api-access-hqw2c\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-kube-api-access-hqw2c\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.037440 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037402 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-data\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.037440 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037427 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b60600-b26d-4594-82aa-0f45f55b8907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b60600-b26d-4594-82aa-0f45f55b8907-istiod-ca-cert\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.037726 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037623 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-credential-socket\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.037726 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037685 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-certs\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.037726 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037713 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-token\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-token\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.037876 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037734 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-data" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:06.037876 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037780 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-envoy\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.037876 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037854 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-podinfo\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.038024 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037907 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-socket\") pod \"f9b60600-b26d-4594-82aa-0f45f55b8907\" (UID: \"f9b60600-b26d-4594-82aa-0f45f55b8907\") " Oct 31 04:00:06.038073 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037816 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b60600-b26d-4594-82aa-0f45f55b8907-istiod-ca-cert" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:06.038134 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037920 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-credential-socket" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:06.038134 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.037953 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-certs" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:06.038134 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.038123 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-socket" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:06.038428 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.038407 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.038687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.038433 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.038687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.038447 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b60600-b26d-4594-82aa-0f45f55b8907-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.038687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.038459 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.038687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.038472 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.039883 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.039860 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-kube-api-access-hqw2c" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "kube-api-access-hqw2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:06.040008 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.039991 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-token" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:06.040141 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.040118 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-podinfo" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:06.040454 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.040432 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-envoy" pod "f9b60600-b26d-4594-82aa-0f45f55b8907" (UID: "f9b60600-b26d-4594-82aa-0f45f55b8907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:06.139179 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.139142 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqw2c\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-kube-api-access-hqw2c\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.139179 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.139181 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.139411 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.139198 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.139411 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.139212 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b60600-b26d-4594-82aa-0f45f55b8907-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.252595 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.252559 4934 generic.go:356] "Generic (PLEG): container finished" podID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerID="8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe" exitCode=0 Oct 31 04:00:06.252803 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.252607 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" Oct 31 04:00:06.252803 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.252642 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" event={"ID":"f9b60600-b26d-4594-82aa-0f45f55b8907","Type":"ContainerDied","Data":"8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe"} Oct 31 04:00:06.252803 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.252703 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq" event={"ID":"f9b60600-b26d-4594-82aa-0f45f55b8907","Type":"ContainerDied","Data":"a28c2d51aba99a99f28b9751f7df8359a7dd8b1a5336560a8871addf021fb1f0"} Oct 31 04:00:06.252803 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.252716 4934 scope.go:122] "RemoveContainer" containerID="8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe" Oct 31 04:00:06.262817 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.262797 4934 scope.go:122] "RemoveContainer" containerID="8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe" Oct 31 04:00:06.263118 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:00:06.263079 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe\": container with ID starting with 8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe not found: ID does not exist" containerID="8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe" Oct 31 04:00:06.263207 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.263130 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe"} err="failed to get container status \"8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe\": rpc error: code = NotFound desc = could not find container \"8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe\": container with ID starting with 8dc816ccf82f3f5f82448a8ff08814bef0ee6076188cf87f8e75f6f0c35d2ffe not found: ID does not exist" Oct 31 04:00:06.417507 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.417473 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq"] Oct 31 04:00:06.425952 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.425926 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-qdqpq"] Oct 31 04:00:06.821330 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:06.821300 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" path="/var/lib/kubelet/pods/f9b60600-b26d-4594-82aa-0f45f55b8907/volumes" Oct 31 04:00:11.711769 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:11.711732 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6"] Oct 31 04:00:11.712256 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:11.711928 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" containerID="cri-o://52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195" gracePeriod=30 Oct 31 04:00:13.421301 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.421267 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n"] Oct 31 04:00:13.421700 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.421458 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" podUID="9da424b7-886e-4ace-985b-d6f5f4b631d0" containerName="maas-api" containerID="cri-o://723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f" gracePeriod=30 Oct 31 04:00:13.709031 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.709001 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 04:00:13.812873 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.811696 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da424b7-886e-4ace-985b-d6f5f4b631d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da424b7-886e-4ace-985b-d6f5f4b631d0-openshift-service-ca\") pod \"9da424b7-886e-4ace-985b-d6f5f4b631d0\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " Oct 31 04:00:13.812873 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.811945 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da424b7-886e-4ace-985b-d6f5f4b631d0-kube-api-access-9xbjc\" (UniqueName: \"kubernetes.io/projected/9da424b7-886e-4ace-985b-d6f5f4b631d0-kube-api-access-9xbjc\") pod \"9da424b7-886e-4ace-985b-d6f5f4b631d0\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " Oct 31 04:00:13.812873 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.811995 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls\") pod \"9da424b7-886e-4ace-985b-d6f5f4b631d0\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " Oct 31 04:00:13.812873 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.812028 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-metrics-certs\") pod \"9da424b7-886e-4ace-985b-d6f5f4b631d0\" (UID: \"9da424b7-886e-4ace-985b-d6f5f4b631d0\") " Oct 31 04:00:13.814040 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.813911 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da424b7-886e-4ace-985b-d6f5f4b631d0-openshift-service-ca" pod "9da424b7-886e-4ace-985b-d6f5f4b631d0" (UID: "9da424b7-886e-4ace-985b-d6f5f4b631d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:13.821777 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.820794 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da424b7-886e-4ace-985b-d6f5f4b631d0-kube-api-access-9xbjc" pod "9da424b7-886e-4ace-985b-d6f5f4b631d0" (UID: "9da424b7-886e-4ace-985b-d6f5f4b631d0"). InnerVolumeSpecName "kube-api-access-9xbjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:13.821777 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.820908 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-metrics-certs" pod "9da424b7-886e-4ace-985b-d6f5f4b631d0" (UID: "9da424b7-886e-4ace-985b-d6f5f4b631d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:13.821777 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.821303 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls" pod "9da424b7-886e-4ace-985b-d6f5f4b631d0" (UID: "9da424b7-886e-4ace-985b-d6f5f4b631d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:13.913506 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.913475 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbjc\" (UniqueName: \"kubernetes.io/projected/9da424b7-886e-4ace-985b-d6f5f4b631d0-kube-api-access-9xbjc\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:13.913506 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.913502 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:13.913506 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.913512 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da424b7-886e-4ace-985b-d6f5f4b631d0-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:13.913733 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:13.913522 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da424b7-886e-4ace-985b-d6f5f4b631d0-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.293269 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.293214 4934 generic.go:356] "Generic (PLEG): container finished" podID="9da424b7-886e-4ace-985b-d6f5f4b631d0" containerID="723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f" exitCode=0 Oct 31 04:00:14.293457 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.293282 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" Oct 31 04:00:14.293457 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.293306 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" event={"ID":"9da424b7-886e-4ace-985b-d6f5f4b631d0","Type":"ContainerDied","Data":"723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f"} Oct 31 04:00:14.293457 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.293389 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n" event={"ID":"9da424b7-886e-4ace-985b-d6f5f4b631d0","Type":"ContainerDied","Data":"80ceeb94cbdd368dfe8ae80373d17f8253ace97f941ee524739d06ba2927fe61"} Oct 31 04:00:14.293457 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.293418 4934 scope.go:122] "RemoveContainer" containerID="723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f" Oct 31 04:00:14.303828 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.303807 4934 scope.go:122] "RemoveContainer" containerID="723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f" Oct 31 04:00:14.304110 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:00:14.304092 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f\": container with ID starting with 723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f not found: ID does not exist" containerID="723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f" Oct 31 04:00:14.304153 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.304118 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f"} err="failed to get container status \"723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f\": rpc error: code = NotFound desc = could not find container \"723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f\": container with ID starting with 723b630fc286299e4302ffa4648d2956d20010f6add2bea5b4375e3963f4462f not found: ID does not exist" Oct 31 04:00:14.457217 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.457181 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n"] Oct 31 04:00:14.463900 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.463872 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-dec4a0-997d995f4-7wf8n"] Oct 31 04:00:14.820359 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:14.820331 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da424b7-886e-4ace-985b-d6f5f4b631d0" path="/var/lib/kubelet/pods/9da424b7-886e-4ace-985b-d6f5f4b631d0/volumes" Oct 31 04:00:16.968449 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:16.968424 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 04:00:17.145530 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145495 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-credential-socket\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.145686 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145559 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istiod-ca-cert\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.145686 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145594 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-data\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.145686 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145639 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-envoy\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.145862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145687 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-podinfo\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.145862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145748 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-kube-api-access-zfh6w\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-kube-api-access-zfh6w\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.145862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145777 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-token\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-token\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.145862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145790 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-credential-socket" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:17.146063 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145866 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-certs\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.146063 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145904 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-socket\") pod \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\" (UID: \"5682ca43-69b2-4c07-b89e-4b5f062f6c67\") " Oct 31 04:00:17.146063 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.145997 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istiod-ca-cert" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:17.146284 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.146209 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-data" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:17.146348 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.146286 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-socket" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:17.146444 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.146420 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.146444 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.146446 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.146612 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.146461 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.146612 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.146473 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.146723 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.146652 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-certs" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:17.148895 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.148868 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-podinfo" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:17.149353 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.149329 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-envoy" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:17.149353 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.149337 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-token" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:17.150178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.150143 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-kube-api-access-zfh6w" pod "5682ca43-69b2-4c07-b89e-4b5f062f6c67" (UID: "5682ca43-69b2-4c07-b89e-4b5f062f6c67"). InnerVolumeSpecName "kube-api-access-zfh6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:17.247480 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.247451 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.247480 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.247475 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.247648 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.247487 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfh6w\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-kube-api-access-zfh6w\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.247648 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.247497 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5682ca43-69b2-4c07-b89e-4b5f062f6c67-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.247648 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.247504 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5682ca43-69b2-4c07-b89e-4b5f062f6c67-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:17.312806 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.312773 4934 generic.go:356] "Generic (PLEG): container finished" podID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerID="52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195" exitCode=0 Oct 31 04:00:17.312986 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.312838 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" Oct 31 04:00:17.312986 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.312859 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" event={"ID":"5682ca43-69b2-4c07-b89e-4b5f062f6c67","Type":"ContainerDied","Data":"52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195"} Oct 31 04:00:17.312986 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.312929 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6" event={"ID":"5682ca43-69b2-4c07-b89e-4b5f062f6c67","Type":"ContainerDied","Data":"645e23bbddf3b74b7983cc9613fd7c8d06920445e53110015e9cfc9669e9db8b"} Oct 31 04:00:17.312986 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.312964 4934 scope.go:122] "RemoveContainer" containerID="52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195" Oct 31 04:00:17.322514 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.322493 4934 scope.go:122] "RemoveContainer" containerID="b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4" Oct 31 04:00:17.330904 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.330881 4934 scope.go:122] "RemoveContainer" containerID="52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195" Oct 31 04:00:17.331176 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:00:17.331156 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195\": container with ID starting with 52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195 not found: ID does not exist" containerID="52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195" Oct 31 04:00:17.331239 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.331186 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195"} err="failed to get container status \"52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195\": rpc error: code = NotFound desc = could not find container \"52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195\": container with ID starting with 52d1dc6629b391a8c7be5d081e16262f2b9692070471bab7d5c3f73fe60cf195 not found: ID does not exist" Oct 31 04:00:17.331239 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.331204 4934 scope.go:122] "RemoveContainer" containerID="b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4" Oct 31 04:00:17.331477 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:00:17.331462 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4\": container with ID starting with b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4 not found: ID does not exist" containerID="b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4" Oct 31 04:00:17.331516 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.331482 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4"} err="failed to get container status \"b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4\": rpc error: code = NotFound desc = could not find container \"b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4\": container with ID starting with b7d93513174ee8b165c466020de206ff4b039866d1865aee9ce613a4123736d4 not found: ID does not exist" Oct 31 04:00:17.477642 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.477564 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6"] Oct 31 04:00:17.481961 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:17.481936 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-phzw6"] Oct 31 04:00:18.820704 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:18.820672 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" path="/var/lib/kubelet/pods/5682ca43-69b2-4c07-b89e-4b5f062f6c67/volumes" Oct 31 04:00:19.162731 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:00:19.162657 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:00:24.226426 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:24.226393 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz"] Oct 31 04:00:24.226855 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:24.226608 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" containerID="cri-o://410ebc613723e652d6d568e4f7d9c34b7133c09c5b237141072577ceb4a0adc5" gracePeriod=30 Oct 31 04:00:29.375095 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.375060 4934 generic.go:356] "Generic (PLEG): container finished" podID="ffe0bb92-7885-42de-bb32-9e3957822804" containerID="410ebc613723e652d6d568e4f7d9c34b7133c09c5b237141072577ceb4a0adc5" exitCode=0 Oct 31 04:00:29.375483 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.375114 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" event={"ID":"ffe0bb92-7885-42de-bb32-9e3957822804","Type":"ContainerDied","Data":"410ebc613723e652d6d568e4f7d9c34b7133c09c5b237141072577ceb4a0adc5"} Oct 31 04:00:29.482622 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.482595 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 04:00:29.562248 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562201 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-certs\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562399 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562274 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-socket\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562399 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562307 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-istio-token\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-istio-token\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562399 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562337 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-kube-api-access-zrhhn\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-kube-api-access-zrhhn\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562399 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562379 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-data\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562399 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562399 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-credential-socket\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562685 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562420 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-envoy\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562685 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562456 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe0bb92-7885-42de-bb32-9e3957822804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe0bb92-7885-42de-bb32-9e3957822804-istiod-ca-cert\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562685 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562490 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffe0bb92-7885-42de-bb32-9e3957822804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe0bb92-7885-42de-bb32-9e3957822804-istio-podinfo\") pod \"ffe0bb92-7885-42de-bb32-9e3957822804\" (UID: \"ffe0bb92-7885-42de-bb32-9e3957822804\") " Oct 31 04:00:29.562685 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562493 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-certs" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:29.562685 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562602 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-socket" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:29.562921 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562699 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-data" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:29.562921 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562719 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-credential-socket" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:29.563014 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562917 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.563014 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562942 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.563014 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562958 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.563014 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562971 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.563014 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.562938 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe0bb92-7885-42de-bb32-9e3957822804-istiod-ca-cert" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:29.564685 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.564656 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-istio-token" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:29.564958 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.564932 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffe0bb92-7885-42de-bb32-9e3957822804-istio-podinfo" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:29.565114 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.565096 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-envoy" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:29.565114 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.565093 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-kube-api-access-zrhhn" pod "ffe0bb92-7885-42de-bb32-9e3957822804" (UID: "ffe0bb92-7885-42de-bb32-9e3957822804"). InnerVolumeSpecName "kube-api-access-zrhhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:29.664425 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.664390 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.664425 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.664421 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrhhn\" (UniqueName: \"kubernetes.io/projected/ffe0bb92-7885-42de-bb32-9e3957822804-kube-api-access-zrhhn\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.664425 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.664434 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe0bb92-7885-42de-bb32-9e3957822804-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.664700 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.664442 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe0bb92-7885-42de-bb32-9e3957822804-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:29.664700 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:29.664455 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe0bb92-7885-42de-bb32-9e3957822804-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:30.381352 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:30.381320 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" Oct 31 04:00:30.381352 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:30.381333 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz" event={"ID":"ffe0bb92-7885-42de-bb32-9e3957822804","Type":"ContainerDied","Data":"7c952fdbeee37648c73a132ce2fcb8ba9194565b5e5aadbe2ac6cbfb22815db5"} Oct 31 04:00:30.381874 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:30.381388 4934 scope.go:122] "RemoveContainer" containerID="410ebc613723e652d6d568e4f7d9c34b7133c09c5b237141072577ceb4a0adc5" Oct 31 04:00:30.595872 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:30.595838 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz"] Oct 31 04:00:30.646274 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:30.646175 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-7t9jz"] Oct 31 04:00:30.823568 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:30.823533 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" path="/var/lib/kubelet/pods/ffe0bb92-7885-42de-bb32-9e3957822804/volumes" Oct 31 04:00:49.167121 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:00:49.167034 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:00:54.258436 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.258400 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr"] Oct 31 04:00:54.258884 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.258598 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" podUID="ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" containerName="maas-api" containerID="cri-o://2a17597e8612c3f2130578b8955bc8d3a14cdf8e4f8495d64e39d81223070864" gracePeriod=30 Oct 31 04:00:54.491555 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.491518 4934 generic.go:356] "Generic (PLEG): container finished" podID="ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" containerID="2a17597e8612c3f2130578b8955bc8d3a14cdf8e4f8495d64e39d81223070864" exitCode=0 Oct 31 04:00:54.491715 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.491561 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" event={"ID":"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae","Type":"ContainerDied","Data":"2a17597e8612c3f2130578b8955bc8d3a14cdf8e4f8495d64e39d81223070864"} Oct 31 04:00:54.513496 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.513430 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 04:00:54.613491 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.613458 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-kube-api-access-scqhw\" (UniqueName: \"kubernetes.io/projected/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-kube-api-access-scqhw\") pod \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " Oct 31 04:00:54.613687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.613514 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls\") pod \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " Oct 31 04:00:54.613687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.613637 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-openshift-service-ca\") pod \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " Oct 31 04:00:54.613687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.613665 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-metrics-certs\") pod \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\" (UID: \"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae\") " Oct 31 04:00:54.614007 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.613984 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-openshift-service-ca" pod "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" (UID: "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:54.615892 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.615871 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-metrics-certs" pod "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" (UID: "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:54.616295 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.616277 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls" pod "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" (UID: "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:54.616369 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.616318 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-kube-api-access-scqhw" pod "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" (UID: "ed1fd977-8cf4-4938-8e6d-b1c2fb61faae"). InnerVolumeSpecName "kube-api-access-scqhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:54.714536 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.714510 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:54.714536 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.714536 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:54.714729 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.714545 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scqhw\" (UniqueName: \"kubernetes.io/projected/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-kube-api-access-scqhw\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:54.714729 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:54.714554 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:55.496615 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:55.496576 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" Oct 31 04:00:55.497059 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:55.496607 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr" event={"ID":"ed1fd977-8cf4-4938-8e6d-b1c2fb61faae","Type":"ContainerDied","Data":"3378c49606f484097fcb62193644a532fc8eced963ffbbd69bf6064de353b3ca"} Oct 31 04:00:55.497059 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:55.496672 4934 scope.go:122] "RemoveContainer" containerID="2a17597e8612c3f2130578b8955bc8d3a14cdf8e4f8495d64e39d81223070864" Oct 31 04:00:55.660652 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:55.660614 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr"] Oct 31 04:00:55.710565 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:55.710527 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6de37a-7bd88bf5bc-gq4kr"] Oct 31 04:00:56.820156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:00:56.820121 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" path="/var/lib/kubelet/pods/ed1fd977-8cf4-4938-8e6d-b1c2fb61faae/volumes" Oct 31 04:01:03.762582 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:03.762547 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn"] Oct 31 04:01:03.762996 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:03.762749 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" containerID="cri-o://f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1" gracePeriod=30 Oct 31 04:01:09.005024 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.004995 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 04:01:09.152536 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152504 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-envoy\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.152536 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152539 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-certs\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.152791 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152569 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-token\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-token\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.152791 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152634 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-credential-socket\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.152791 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152664 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-data\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.152791 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152726 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-podinfo\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.152791 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152758 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istiod-ca-cert\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.153057 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152812 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-socket\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.153057 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152836 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-certs" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:09.153057 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152905 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-kube-api-access-d4qw8\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-kube-api-access-d4qw8\") pod \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\" (UID: \"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a\") " Oct 31 04:01:09.153057 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152925 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-credential-socket" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:09.153057 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.152999 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-data" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:09.153494 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.153244 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-socket" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:09.153494 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.153316 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.153494 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.153313 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istiod-ca-cert" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:01:09.153494 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.153341 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.153494 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.153367 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.155069 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.155017 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-envoy" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:09.155184 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.155093 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-token" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:09.155413 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.155381 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-podinfo" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:01:09.155538 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.155456 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-kube-api-access-d4qw8" pod "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" (UID: "98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a"). InnerVolumeSpecName "kube-api-access-d4qw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:09.254940 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.254899 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.254940 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.254939 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.255164 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.254954 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.255164 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.255013 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qw8\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-kube-api-access-d4qw8\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.255164 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.255027 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.255164 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.255042 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.569087 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.569049 4934 generic.go:356] "Generic (PLEG): container finished" podID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerID="f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1" exitCode=0 Oct 31 04:01:09.569298 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.569111 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" Oct 31 04:01:09.569298 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.569135 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" event={"ID":"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a","Type":"ContainerDied","Data":"f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1"} Oct 31 04:01:09.569298 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.569185 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn" event={"ID":"98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a","Type":"ContainerDied","Data":"dc8a961c285c626bb2aea4dd17c567600de57ae09d545cae91debbf604de7600"} Oct 31 04:01:09.569298 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.569207 4934 scope.go:122] "RemoveContainer" containerID="f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1" Oct 31 04:01:09.582434 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.582384 4934 scope.go:122] "RemoveContainer" containerID="85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483" Oct 31 04:01:09.591706 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.591683 4934 scope.go:122] "RemoveContainer" containerID="f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1" Oct 31 04:01:09.591983 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:01:09.591962 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1\": container with ID starting with f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1 not found: ID does not exist" containerID="f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1" Oct 31 04:01:09.592060 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.591993 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1"} err="failed to get container status \"f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1\": rpc error: code = NotFound desc = could not find container \"f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1\": container with ID starting with f1a54f3499ed0c518a815ecb2c5f564c1081ca4108a3470b49c95f55712f7ad1 not found: ID does not exist" Oct 31 04:01:09.592060 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.592009 4934 scope.go:122] "RemoveContainer" containerID="85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483" Oct 31 04:01:09.592262 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:01:09.592243 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483\": container with ID starting with 85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483 not found: ID does not exist" containerID="85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483" Oct 31 04:01:09.592329 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.592270 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483"} err="failed to get container status \"85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483\": rpc error: code = NotFound desc = could not find container \"85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483\": container with ID starting with 85a7a0af463ee4b3d9905e664a39498c2edc2e9ffa40afcfafdba849f95b1483 not found: ID does not exist" Oct 31 04:01:09.735346 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.735315 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn"] Oct 31 04:01:09.785093 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:09.785061 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6de37a-openshift-default-54f647bb45-bzfjn"] Oct 31 04:01:10.820054 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:01:10.820016 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" path="/var/lib/kubelet/pods/98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a/volumes" Oct 31 04:01:19.161870 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:01:19.161835 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:01:49.166067 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:01:49.166029 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:02:02.282969 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.282932 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5"] Oct 31 04:02:02.288255 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.287964 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" podUID="f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" containerName="maas-api" containerID="cri-o://6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee" gracePeriod=30 Oct 31 04:02:02.545111 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.545054 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 04:02:02.635083 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.635046 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-openshift-service-ca\") pod \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " Oct 31 04:02:02.635199 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.635139 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls\") pod \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " Oct 31 04:02:02.635199 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.635178 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-kube-api-access-vc22v\" (UniqueName: \"kubernetes.io/projected/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-kube-api-access-vc22v\") pod \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " Oct 31 04:02:02.635199 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.635195 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-metrics-certs\") pod \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\" (UID: \"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2\") " Oct 31 04:02:02.635456 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.635434 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-openshift-service-ca" pod "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" (UID: "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:02.637312 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.637285 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls" pod "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" (UID: "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:02.637312 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.637304 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-metrics-certs" pod "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" (UID: "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:02.637526 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.637506 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-kube-api-access-vc22v" pod "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" (UID: "f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2"). InnerVolumeSpecName "kube-api-access-vc22v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:02.736658 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.736618 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:02.736658 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.736650 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:02.736658 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.736663 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc22v\" (UniqueName: \"kubernetes.io/projected/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-kube-api-access-vc22v\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:02.736890 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.736676 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:02.802512 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.802422 4934 generic.go:356] "Generic (PLEG): container finished" podID="f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" containerID="6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee" exitCode=0 Oct 31 04:02:02.802512 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.802477 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" Oct 31 04:02:02.802693 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.802509 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" event={"ID":"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2","Type":"ContainerDied","Data":"6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee"} Oct 31 04:02:02.802693 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.802566 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5" event={"ID":"f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2","Type":"ContainerDied","Data":"ac95a588a42655e6c1a7f3f2a89224558c3f4a73e9fc2b95008c618be141adde"} Oct 31 04:02:02.802693 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.802588 4934 scope.go:122] "RemoveContainer" containerID="6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee" Oct 31 04:02:02.812466 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.812449 4934 scope.go:122] "RemoveContainer" containerID="6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee" Oct 31 04:02:02.812734 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:02:02.812716 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee\": container with ID starting with 6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee not found: ID does not exist" containerID="6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee" Oct 31 04:02:02.812778 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.812742 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee"} err="failed to get container status \"6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee\": rpc error: code = NotFound desc = could not find container \"6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee\": container with ID starting with 6fa064e81055823dd729e9e41a0f0fa1693c5e0cbf34c9b2abefed5f3c55e5ee not found: ID does not exist" Oct 31 04:02:02.967641 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.967606 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5"] Oct 31 04:02:02.975094 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:02.975058 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-14cb5a-6864ff8d5d-snnf5"] Oct 31 04:02:04.818962 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:04.818930 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" path="/var/lib/kubelet/pods/f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2/volumes" Oct 31 04:02:19.162431 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:02:19.162383 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:02:19.228243 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228199 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-82hxz"] Oct 31 04:02:19.228617 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228601 4934 cpu_manager.go:414] "containerMap: removing container" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228620 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228631 4934 cpu_manager.go:414] "containerMap: removing container" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228638 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228656 4934 cpu_manager.go:414] "containerMap: removing container" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228665 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228677 4934 cpu_manager.go:414] "containerMap: removing container" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228684 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228693 4934 cpu_manager.go:414] "containerMap: removing container" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" Oct 31 04:02:19.228703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228699 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228708 4934 cpu_manager.go:414] "containerMap: removing container" podUID="ac2d9504-ef57-4231-bfa3-67ae4a8db56e" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228715 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2d9504-ef57-4231-bfa3-67ae4a8db56e" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228728 4934 cpu_manager.go:414] "containerMap: removing container" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228735 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228748 4934 cpu_manager.go:414] "containerMap: removing container" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228755 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228767 4934 cpu_manager.go:414] "containerMap: removing container" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228774 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228785 4934 cpu_manager.go:414] "containerMap: removing container" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228792 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228802 4934 cpu_manager.go:414] "containerMap: removing container" podUID="762ead81-eb44-4059-a19e-d6704ea7a111" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228809 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762ead81-eb44-4059-a19e-d6704ea7a111" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228820 4934 cpu_manager.go:414] "containerMap: removing container" podUID="4c9933ee-1180-4228-a595-440a790a7d20" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228827 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9933ee-1180-4228-a595-440a790a7d20" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228840 4934 cpu_manager.go:414] "containerMap: removing container" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228847 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228858 4934 cpu_manager.go:414] "containerMap: removing container" podUID="febda22b-0402-4a68-a578-4202936a45b7" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228865 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febda22b-0402-4a68-a578-4202936a45b7" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228875 4934 cpu_manager.go:414] "containerMap: removing container" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228882 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228891 4934 cpu_manager.go:414] "containerMap: removing container" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228897 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228905 4934 cpu_manager.go:414] "containerMap: removing container" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228912 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228929 4934 cpu_manager.go:414] "containerMap: removing container" podUID="f05f22f9-4f34-4f85-836b-6d895a800855" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228936 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05f22f9-4f34-4f85-836b-6d895a800855" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228945 4934 cpu_manager.go:414] "containerMap: removing container" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228952 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228963 4934 cpu_manager.go:414] "containerMap: removing container" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228970 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228981 4934 cpu_manager.go:414] "containerMap: removing container" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228988 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.228997 4934 cpu_manager.go:414] "containerMap: removing container" podUID="a0c01170-981e-423d-8529-5ae0076a99de" containerName="test-curl-7165-ea87cd" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229004 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c01170-981e-423d-8529-5ae0076a99de" containerName="test-curl-7165-ea87cd" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229017 4934 cpu_manager.go:414] "containerMap: removing container" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229024 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229036 4934 cpu_manager.go:414] "containerMap: removing container" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229042 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229055 4934 cpu_manager.go:414] "containerMap: removing container" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229062 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229071 4934 cpu_manager.go:414] "containerMap: removing container" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229077 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229086 4934 cpu_manager.go:414] "containerMap: removing container" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229093 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229104 4934 cpu_manager.go:414] "containerMap: removing container" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229110 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229121 4934 cpu_manager.go:414] "containerMap: removing container" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229129 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229139 4934 cpu_manager.go:414] "containerMap: removing container" podUID="9da424b7-886e-4ace-985b-d6f5f4b631d0" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229146 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da424b7-886e-4ace-985b-d6f5f4b631d0" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229154 4934 cpu_manager.go:414] "containerMap: removing container" podUID="c44742f8-e136-48da-95c6-8cd88b73e76f" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229161 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44742f8-e136-48da-95c6-8cd88b73e76f" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229170 4934 cpu_manager.go:414] "containerMap: removing container" podUID="840e5980-4118-4f8d-988b-2920003c251c" containerName="maas-api" Oct 31 04:02:19.229156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229177 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="840e5980-4118-4f8d-988b-2920003c251c" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229188 4934 cpu_manager.go:414] "containerMap: removing container" podUID="f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229194 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229205 4934 cpu_manager.go:414] "containerMap: removing container" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229213 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229222 4934 cpu_manager.go:414] "containerMap: removing container" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229247 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229259 4934 cpu_manager.go:414] "containerMap: removing container" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229265 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229278 4934 cpu_manager.go:414] "containerMap: removing container" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229286 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229296 4934 cpu_manager.go:414] "containerMap: removing container" podUID="ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229302 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229399 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8088fc-01d4-4ce9-85f4-32fd06caf3f8" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229415 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edb95a7-fa74-487f-ab8a-7658305c77ee" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229424 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b60600-b26d-4594-82aa-0f45f55b8907" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229433 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229440 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b7182b-43bf-4208-8836-ce6eb34a5a14" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229451 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fa25de-5c01-4e68-b69d-91699978b69f" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229459 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229467 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da424b7-886e-4ace-985b-d6f5f4b631d0" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229475 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44742f8-e136-48da-95c6-8cd88b73e76f" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229482 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="840e5980-4118-4f8d-988b-2920003c251c" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229491 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="f731aec1-c19b-4e31-a5dc-0b7bbc8d55f2" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229499 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229507 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="17accd80-6cc2-4d3d-8e29-1db30b44852d" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229514 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229524 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2375106-1e2f-44a8-a7bf-dda13216fe2a" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229532 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1fd977-8cf4-4938-8e6d-b1c2fb61faae" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229540 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f4a9ca-e8db-4587-9a1a-08471c508625" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229547 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e73b68-b1ec-41c2-81bc-160ad40bce81" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229556 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229564 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d07ca2b-173c-4293-bc8f-2f8e5bf95e14" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229573 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2d9504-ef57-4231-bfa3-67ae4a8db56e" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229584 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229593 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06443ca-871f-413c-885f-200727268117" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229602 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="98adbaed-6f0f-4eeb-8e5a-1cd9b8bd293a" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229612 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2e050d-524e-4844-ba8c-9e4955f9ff7c" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229618 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="762ead81-eb44-4059-a19e-d6704ea7a111" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229626 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9933ee-1180-4228-a595-440a790a7d20" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229634 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="520f796a-3221-4eaf-b27f-0b8c643476f1" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229642 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="febda22b-0402-4a68-a578-4202936a45b7" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229651 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="5682ca43-69b2-4c07-b89e-4b5f062f6c67" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229660 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37a6959-4519-4d12-aa29-86651e36705b" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229671 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05f22f9-4f34-4f85-836b-6d895a800855" containerName="maas-api" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229678 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="365b8a0d-2476-4e6f-95fa-ddcbad399dca" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229686 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a4cda6-f9c8-441c-ad18-c335c59fb134" containerName="main" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229694 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe0bb92-7885-42de-bb32-9e3957822804" containerName="istio-proxy" Oct 31 04:02:19.231406 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.229702 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c01170-981e-423d-8529-5ae0076a99de" containerName="test-curl-7165-ea87cd" Oct 31 04:02:19.234446 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.234426 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.291092 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.291066 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 04:02:19.341387 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.341364 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2xtxp\"" Oct 31 04:02:19.341491 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.341415 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 04:02:19.341626 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.341611 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 04:02:19.342809 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.342782 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-82hxz"] Oct 31 04:02:19.375122 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.375101 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-webhook-cert\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.375212 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.375134 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-metrics-certs\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.375212 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.375154 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vwn\" (UniqueName: \"kubernetes.io/projected/2f88f6ca-1fe3-461a-8f86-010c367d21d9-kube-api-access-s7vwn\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.476310 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.476247 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-webhook-cert\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.476310 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.476293 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-metrics-certs\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.476474 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.476319 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vwn\" (UniqueName: \"kubernetes.io/projected/2f88f6ca-1fe3-461a-8f86-010c367d21d9-kube-api-access-s7vwn\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.478832 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.478811 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-webhook-cert\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.478948 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.478928 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-metrics-certs\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.537926 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.537906 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vwn\" (UniqueName: \"kubernetes.io/projected/2f88f6ca-1fe3-461a-8f86-010c367d21d9-kube-api-access-s7vwn\") pod \"maas-controller-768d4c8b4f-82hxz\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.546703 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.546680 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:19.925014 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.923613 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-82hxz"] Oct 31 04:02:19.927009 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:02:19.926971 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f88f6ca_1fe3_461a_8f86_010c367d21d9.slice/crio-65d8d91820d71b9ac9e812ec08711ce683c93be4193c9ad66c33a3a7145d3470 WatchSource:0}: Error finding container 65d8d91820d71b9ac9e812ec08711ce683c93be4193c9ad66c33a3a7145d3470: Status 404 returned error can't find the container with id 65d8d91820d71b9ac9e812ec08711ce683c93be4193c9ad66c33a3a7145d3470 Oct 31 04:02:19.928761 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:19.928745 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:02:20.907251 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:20.894027 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" event={"ID":"2f88f6ca-1fe3-461a-8f86-010c367d21d9","Type":"ContainerStarted","Data":"65d8d91820d71b9ac9e812ec08711ce683c93be4193c9ad66c33a3a7145d3470"} Oct 31 04:02:23.918430 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:23.918386 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" event={"ID":"2f88f6ca-1fe3-461a-8f86-010c367d21d9","Type":"ContainerStarted","Data":"978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434"} Oct 31 04:02:23.918803 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:23.918583 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:24.039342 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:24.039295 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" podStartSLOduration=2.053315653 podStartE2EDuration="5.039282141s" totalImagesPullingTime="2.985966488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:24.037866317 +0000 UTC m=+2577.356608944" watchObservedRunningTime="2026-10-31 04:02:24.039282141 +0000 UTC m=+2577.358024768" Oct 31 04:02:34.924659 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:34.924629 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:49.170839 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:02:49.170793 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:02:56.976052 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:56.976020 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-82hxz"] Oct 31 04:02:56.976517 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:56.976243 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" podUID="2f88f6ca-1fe3-461a-8f86-010c367d21d9" containerName="manager" containerID="cri-o://978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434" gracePeriod=30 Oct 31 04:02:57.220583 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.220559 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:57.365566 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.365532 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-webhook-cert\") pod \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " Oct 31 04:02:57.365566 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.365568 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-metrics-certs\") pod \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " Oct 31 04:02:57.365832 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.365598 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f88f6ca-1fe3-461a-8f86-010c367d21d9-kube-api-access-s7vwn\" (UniqueName: \"kubernetes.io/projected/2f88f6ca-1fe3-461a-8f86-010c367d21d9-kube-api-access-s7vwn\") pod \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\" (UID: \"2f88f6ca-1fe3-461a-8f86-010c367d21d9\") " Oct 31 04:02:57.367951 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.367923 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f88f6ca-1fe3-461a-8f86-010c367d21d9-kube-api-access-s7vwn" pod "2f88f6ca-1fe3-461a-8f86-010c367d21d9" (UID: "2f88f6ca-1fe3-461a-8f86-010c367d21d9"). InnerVolumeSpecName "kube-api-access-s7vwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:57.368376 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.368355 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-webhook-cert" pod "2f88f6ca-1fe3-461a-8f86-010c367d21d9" (UID: "2f88f6ca-1fe3-461a-8f86-010c367d21d9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:57.368556 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.368537 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-metrics-certs" pod "2f88f6ca-1fe3-461a-8f86-010c367d21d9" (UID: "2f88f6ca-1fe3-461a-8f86-010c367d21d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:57.466574 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.466540 4934 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-webhook-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:57.466574 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.466566 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f88f6ca-1fe3-461a-8f86-010c367d21d9-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:57.466574 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:57.466577 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7vwn\" (UniqueName: \"kubernetes.io/projected/2f88f6ca-1fe3-461a-8f86-010c367d21d9-kube-api-access-s7vwn\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:58.057741 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.057702 4934 generic.go:356] "Generic (PLEG): container finished" podID="2f88f6ca-1fe3-461a-8f86-010c367d21d9" containerID="978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434" exitCode=0 Oct 31 04:02:58.058343 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.057753 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" Oct 31 04:02:58.058343 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.057748 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" event={"ID":"2f88f6ca-1fe3-461a-8f86-010c367d21d9","Type":"ContainerDied","Data":"978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434"} Oct 31 04:02:58.058343 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.057902 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-82hxz" event={"ID":"2f88f6ca-1fe3-461a-8f86-010c367d21d9","Type":"ContainerDied","Data":"65d8d91820d71b9ac9e812ec08711ce683c93be4193c9ad66c33a3a7145d3470"} Oct 31 04:02:58.058343 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.057938 4934 scope.go:122] "RemoveContainer" containerID="978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434" Oct 31 04:02:58.067715 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.067696 4934 scope.go:122] "RemoveContainer" containerID="978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434" Oct 31 04:02:58.067979 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:02:58.067962 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434\": container with ID starting with 978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434 not found: ID does not exist" containerID="978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434" Oct 31 04:02:58.068046 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.067987 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434"} err="failed to get container status \"978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434\": rpc error: code = NotFound desc = could not find container \"978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434\": container with ID starting with 978e82aadd474e1e19f42aeaf29799acf99afba7ae4b681ad5d91c50b3540434 not found: ID does not exist" Oct 31 04:02:58.221358 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.221323 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-82hxz"] Oct 31 04:02:58.271285 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.271251 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-82hxz"] Oct 31 04:02:58.820614 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:58.820580 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f88f6ca-1fe3-461a-8f86-010c367d21d9" path="/var/lib/kubelet/pods/2f88f6ca-1fe3-461a-8f86-010c367d21d9/volumes" Oct 31 04:02:59.618275 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618242 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2k2s9"] Oct 31 04:02:59.618689 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618619 4934 cpu_manager.go:414] "containerMap: removing container" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" Oct 31 04:02:59.618689 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618631 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" Oct 31 04:02:59.618689 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618642 4934 cpu_manager.go:414] "containerMap: removing container" podUID="2f88f6ca-1fe3-461a-8f86-010c367d21d9" containerName="manager" Oct 31 04:02:59.618689 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618648 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f88f6ca-1fe3-461a-8f86-010c367d21d9" containerName="manager" Oct 31 04:02:59.618833 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618719 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c01246-a320-4e53-97f9-f26a2591547c" containerName="istio-proxy" Oct 31 04:02:59.618833 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618727 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f88f6ca-1fe3-461a-8f86-010c367d21d9" containerName="manager" Oct 31 04:02:59.618833 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618735 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="d106d91d-2fbd-4dbb-8b6f-5601d33d95b0" containerName="istio-proxy" Oct 31 04:02:59.618833 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.618741 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a196b3-a77a-414a-9995-b766598757aa" containerName="istio-proxy" Oct 31 04:02:59.621337 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.621320 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.674468 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.674438 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 04:02:59.724450 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.724425 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2xtxp\"" Oct 31 04:02:59.724450 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.724426 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 04:02:59.724664 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.724573 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 04:02:59.726322 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.726300 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2k2s9"] Oct 31 04:02:59.787038 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.787003 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb9a6c42-db24-41bc-a5eb-666e1c180761-metrics-certs\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.787215 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.787118 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb9a6c42-db24-41bc-a5eb-666e1c180761-webhook-cert\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.787215 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.787159 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6v6g\" (UniqueName: \"kubernetes.io/projected/bb9a6c42-db24-41bc-a5eb-666e1c180761-kube-api-access-n6v6g\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.888221 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.888125 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6v6g\" (UniqueName: \"kubernetes.io/projected/bb9a6c42-db24-41bc-a5eb-666e1c180761-kube-api-access-n6v6g\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.888221 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.888205 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb9a6c42-db24-41bc-a5eb-666e1c180761-metrics-certs\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.888496 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.888315 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb9a6c42-db24-41bc-a5eb-666e1c180761-webhook-cert\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.890868 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.890839 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb9a6c42-db24-41bc-a5eb-666e1c180761-metrics-certs\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.890868 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.890862 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb9a6c42-db24-41bc-a5eb-666e1c180761-webhook-cert\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:02:59.949541 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:02:59.949509 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6v6g\" (UniqueName: \"kubernetes.io/projected/bb9a6c42-db24-41bc-a5eb-666e1c180761-kube-api-access-n6v6g\") pod \"maas-controller-768d4c8b4f-2k2s9\" (UID: \"bb9a6c42-db24-41bc-a5eb-666e1c180761\") " pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:03:00.231988 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:03:00.231903 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:03:00.606062 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:03:00.605987 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2k2s9"] Oct 31 04:03:00.608675 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:03:00.608651 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9a6c42_db24_41bc_a5eb_666e1c180761.slice/crio-dcd63aacdfd5101d5dce30079586e49f3ab4d49b59d6989f3b62ffa233b17ed1 WatchSource:0}: Error finding container dcd63aacdfd5101d5dce30079586e49f3ab4d49b59d6989f3b62ffa233b17ed1: Status 404 returned error can't find the container with id dcd63aacdfd5101d5dce30079586e49f3ab4d49b59d6989f3b62ffa233b17ed1 Oct 31 04:03:01.072873 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:03:01.072841 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" event={"ID":"bb9a6c42-db24-41bc-a5eb-666e1c180761","Type":"ContainerStarted","Data":"dcd63aacdfd5101d5dce30079586e49f3ab4d49b59d6989f3b62ffa233b17ed1"} Oct 31 04:03:02.079825 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:03:02.079794 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" event={"ID":"bb9a6c42-db24-41bc-a5eb-666e1c180761","Type":"ContainerStarted","Data":"b4e7a10a2a7284b85297e4f73e036b643bd567049786839a652f05b9ceaa76a5"} Oct 31 04:03:02.080280 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:03:02.079913 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:03:02.200350 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:03:02.200296 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" podStartSLOduration=2.265817998 podStartE2EDuration="3.200280434s" totalImagesPullingTime="934.462436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:02.199557116 +0000 UTC m=+2615.518299744" watchObservedRunningTime="2026-10-31 04:03:02.200280434 +0000 UTC m=+2615.519023062" Oct 31 04:03:13.086534 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:03:13.086505 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-768d4c8b4f-2k2s9" Oct 31 04:03:19.164794 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:03:19.164757 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:03:49.220977 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:03:49.220850 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:04:19.164741 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:04:19.164701 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:04:27.698792 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:04:27.698761 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:04:27.760617 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:04:27.760593 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:04:49.162830 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:04:49.162785 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:05:19.166526 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:05:19.166490 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:05:49.162277 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:05:49.162241 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:06:19.171103 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:06:19.171058 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:06:49.177759 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:06:49.177662 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:07:19.164333 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:07:19.164295 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:07:49.166168 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:07:49.166123 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:08:19.162333 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:08:19.162288 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:08:49.235269 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:08:49.234998 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:09:12.437478 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:12.437431 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:09:12.438038 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:12.437534 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:09:19.171703 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:09:19.171665 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:09:27.725442 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:27.725410 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:09:27.788796 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:27.788769 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:09:49.163495 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:09:49.163396 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:09:56.747361 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.747328 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4"] Oct 31 04:09:56.750899 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.750871 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.804048 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.804015 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-20cd9e-openshift-default-dockercfg-28jwn\"" Oct 31 04:09:56.853676 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.853627 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-socket\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.853676 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.853671 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-podinfo\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.853947 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.853776 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-credential-socket\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.853947 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.853812 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-data\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.854063 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.853993 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-certs\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.854136 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.854064 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-envoy\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.854136 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.854119 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52qs\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-kube-api-access-k52qs\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.854813 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.854144 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-token\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.854813 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.854170 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0278ce-8527-4f03-9973-7838ecdf03b2-istiod-ca-cert\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.855855 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.855828 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4"] Oct 31 04:09:56.954982 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.954945 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-envoy\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955000 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52qs\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-kube-api-access-k52qs\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955024 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-token\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955048 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0278ce-8527-4f03-9973-7838ecdf03b2-istiod-ca-cert\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955103 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-socket\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955128 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-podinfo\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955149 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-credential-socket\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955178 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955170 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-data\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.955616 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955317 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-certs\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.956247 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955905 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-data\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.956247 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955928 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-socket\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.956247 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.955999 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-credential-socket\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.956455 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.956399 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-certs\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.956528 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.956510 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0278ce-8527-4f03-9973-7838ecdf03b2-istiod-ca-cert\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.957687 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.957665 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-envoy\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:56.958532 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:56.958515 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-podinfo\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:57.018953 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:57.018883 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-token\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:57.019518 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:57.019494 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52qs\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-kube-api-access-k52qs\") pod \"e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:57.067440 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:57.067410 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:57.495164 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:57.495143 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4"] Oct 31 04:09:57.497459 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:09:57.497427 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0278ce_8527_4f03_9973_7838ecdf03b2.slice/crio-7332f3ff25738a151b71d713b1efb2d34cc66966e3692bfae28780682ad8eb0a WatchSource:0}: Error finding container 7332f3ff25738a151b71d713b1efb2d34cc66966e3692bfae28780682ad8eb0a: Status 404 returned error can't find the container with id 7332f3ff25738a151b71d713b1efb2d34cc66966e3692bfae28780682ad8eb0a Oct 31 04:09:57.499485 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:57.499454 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:09:57.499572 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:57.499523 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:09:57.499572 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:57.499551 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:09:58.105203 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:58.105160 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" event={"ID":"1c0278ce-8527-4f03-9973-7838ecdf03b2","Type":"ContainerStarted","Data":"4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0"} Oct 31 04:09:58.105622 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:58.105249 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" event={"ID":"1c0278ce-8527-4f03-9973-7838ecdf03b2","Type":"ContainerStarted","Data":"7332f3ff25738a151b71d713b1efb2d34cc66966e3692bfae28780682ad8eb0a"} Oct 31 04:09:58.227419 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:58.227360 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" podStartSLOduration=2.22734311 podStartE2EDuration="2.22734311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:58.226030855 +0000 UTC m=+3031.544773485" watchObservedRunningTime="2026-10-31 04:09:58.22734311 +0000 UTC m=+3031.546085811" Oct 31 04:09:59.068240 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:59.068182 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:59.074024 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:59.073995 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:59.110656 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:59.110627 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:09:59.111565 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:09:59.111543 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:10:19.190117 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:10:19.189986 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:10:25.238463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:25.238427 4934 generic.go:356] "Generic (PLEG): container finished" podID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerID="4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0" exitCode=0 Oct 31 04:10:25.239050 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:25.238512 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" event={"ID":"1c0278ce-8527-4f03-9973-7838ecdf03b2","Type":"ContainerDied","Data":"4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0"} Oct 31 04:10:25.239050 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:25.238914 4934 scope.go:122] "RemoveContainer" containerID="4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0" Oct 31 04:10:25.240237 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:25.240199 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:10:25.240325 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:25.240309 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:10:25.240362 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:25.240351 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:10:26.244829 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:26.244794 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" event={"ID":"1c0278ce-8527-4f03-9973-7838ecdf03b2","Type":"ContainerStarted","Data":"ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f"} Oct 31 04:10:27.079186 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:27.076093 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:10:27.079186 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:27.076313 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:10:27.079186 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:27.076393 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 04:10:28.070637 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:28.070598 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:10:28.071281 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:28.071248 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 04:10:29.070254 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.069683 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:10:29.070254 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.069750 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 04:10:29.111433 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.111389 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:10:29.344591 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.344515 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l"] Oct 31 04:10:29.348008 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.347991 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.401107 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.401076 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-20cd9e\"/\"kube-root-ca.crt\"" Oct 31 04:10:29.403655 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.403631 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78521f4f-2f40-4d1f-8490-d0764d4b7604-tls-certs\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.403775 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.403665 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsngp\" (UniqueName: \"kubernetes.io/projected/78521f4f-2f40-4d1f-8490-d0764d4b7604-kube-api-access-lsngp\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.403775 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.403701 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-home\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.403920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.403889 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-dshm\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.404046 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.404016 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-model-cache\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.404174 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.404154 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-tmp-dir\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.451017 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.450974 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-20cd9e\"/\"openshift-service-ca.crt\"" Oct 31 04:10:29.451017 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.451004 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-20cd9e\"/\"default-dockercfg-4pvd2\"" Oct 31 04:10:29.451337 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.451004 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-20cd9e\"/\"simulator-single-main-20cd9e-kserve-self-signed-certs\"" Oct 31 04:10:29.452545 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.452519 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l"] Oct 31 04:10:29.504649 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.504613 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-model-cache\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.504866 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.504681 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-tmp-dir\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.504866 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.504720 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78521f4f-2f40-4d1f-8490-d0764d4b7604-tls-certs\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.504980 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.504890 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsngp\" (UniqueName: \"kubernetes.io/projected/78521f4f-2f40-4d1f-8490-d0764d4b7604-kube-api-access-lsngp\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.505035 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.504981 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-home\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.505086 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.505039 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-dshm\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.505589 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.505541 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-model-cache\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.505711 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.505609 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-home\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.505711 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.505698 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-tmp-dir\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.507731 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.507698 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-dshm\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.511823 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.511799 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78521f4f-2f40-4d1f-8490-d0764d4b7604-tls-certs\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.581313 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.581274 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsngp\" (UniqueName: \"kubernetes.io/projected/78521f4f-2f40-4d1f-8490-d0764d4b7604-kube-api-access-lsngp\") pod \"simulator-single-main-20cd9e-kserve-68d5c49557-jr42l\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:29.701747 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:29.701646 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:30.075256 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:30.070694 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:10:30.075256 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:30.070759 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 04:10:30.097683 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:30.097655 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l"] Oct 31 04:10:30.100033 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:10:30.100005 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78521f4f_2f40_4d1f_8490_d0764d4b7604.slice/crio-66f0fbf5bccf30b1b337ff5bc8acac3e7f44c092a271ce44f0604fc6d81b9a59 WatchSource:0}: Error finding container 66f0fbf5bccf30b1b337ff5bc8acac3e7f44c092a271ce44f0604fc6d81b9a59: Status 404 returned error can't find the container with id 66f0fbf5bccf30b1b337ff5bc8acac3e7f44c092a271ce44f0604fc6d81b9a59 Oct 31 04:10:30.289125 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:30.289090 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" event={"ID":"78521f4f-2f40-4d1f-8490-d0764d4b7604","Type":"ContainerStarted","Data":"80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c"} Oct 31 04:10:30.289125 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:30.289130 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" event={"ID":"78521f4f-2f40-4d1f-8490-d0764d4b7604","Type":"ContainerStarted","Data":"66f0fbf5bccf30b1b337ff5bc8acac3e7f44c092a271ce44f0604fc6d81b9a59"} Oct 31 04:10:30.406426 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:30.406307 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" podStartSLOduration=1.4062847600000001 podStartE2EDuration="1.40628476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:30.405982609 +0000 UTC m=+3063.724725251" watchObservedRunningTime="2026-10-31 04:10:30.40628476 +0000 UTC m=+3063.725027388" Oct 31 04:10:30.702756 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:30.702668 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:31.078255 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:31.073668 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 04:10:31.078255 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:31.073730 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 04:10:32.072808 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:32.072773 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:10:32.073975 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:32.073950 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:10:39.702557 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:39.702507 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:39.715711 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:39.715687 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:40.348850 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:10:40.348819 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:10:49.162848 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:10:49.162810 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:11:19.163785 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:11:19.163745 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:11:31.113969 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:31.113928 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l"] Oct 31 04:11:46.140426 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.140378 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" podUID="78521f4f-2f40-4d1f-8490-d0764d4b7604" containerName="main" containerID="cri-o://80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c" gracePeriod=45 Oct 31 04:11:46.382138 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.382111 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:11:46.496065 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.495987 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-home\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-home\") pod \"78521f4f-2f40-4d1f-8490-d0764d4b7604\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " Oct 31 04:11:46.496218 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496110 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78521f4f-2f40-4d1f-8490-d0764d4b7604-kube-api-access-lsngp\" (UniqueName: \"kubernetes.io/projected/78521f4f-2f40-4d1f-8490-d0764d4b7604-kube-api-access-lsngp\") pod \"78521f4f-2f40-4d1f-8490-d0764d4b7604\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " Oct 31 04:11:46.496218 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496149 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78521f4f-2f40-4d1f-8490-d0764d4b7604-tls-certs\" (UniqueName: \"kubernetes.io/secret/78521f4f-2f40-4d1f-8490-d0764d4b7604-tls-certs\") pod \"78521f4f-2f40-4d1f-8490-d0764d4b7604\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " Oct 31 04:11:46.496218 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496200 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-tmp-dir\") pod \"78521f4f-2f40-4d1f-8490-d0764d4b7604\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " Oct 31 04:11:46.496431 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496220 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-home" pod "78521f4f-2f40-4d1f-8490-d0764d4b7604" (UID: "78521f4f-2f40-4d1f-8490-d0764d4b7604"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:46.496431 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496287 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-dshm\") pod \"78521f4f-2f40-4d1f-8490-d0764d4b7604\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " Oct 31 04:11:46.496431 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496322 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-model-cache\") pod \"78521f4f-2f40-4d1f-8490-d0764d4b7604\" (UID: \"78521f4f-2f40-4d1f-8490-d0764d4b7604\") " Oct 31 04:11:46.496604 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496560 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-tmp-dir" pod "78521f4f-2f40-4d1f-8490-d0764d4b7604" (UID: "78521f4f-2f40-4d1f-8490-d0764d4b7604"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:46.496662 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496605 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.496662 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.496630 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-model-cache" pod "78521f4f-2f40-4d1f-8490-d0764d4b7604" (UID: "78521f4f-2f40-4d1f-8490-d0764d4b7604"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:46.498697 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.498665 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-dshm" pod "78521f4f-2f40-4d1f-8490-d0764d4b7604" (UID: "78521f4f-2f40-4d1f-8490-d0764d4b7604"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:46.499132 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.499097 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78521f4f-2f40-4d1f-8490-d0764d4b7604-tls-certs" pod "78521f4f-2f40-4d1f-8490-d0764d4b7604" (UID: "78521f4f-2f40-4d1f-8490-d0764d4b7604"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:46.499387 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.499358 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78521f4f-2f40-4d1f-8490-d0764d4b7604-kube-api-access-lsngp" pod "78521f4f-2f40-4d1f-8490-d0764d4b7604" (UID: "78521f4f-2f40-4d1f-8490-d0764d4b7604"). InnerVolumeSpecName "kube-api-access-lsngp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:46.597830 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.597799 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.597979 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.597834 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsngp\" (UniqueName: \"kubernetes.io/projected/78521f4f-2f40-4d1f-8490-d0764d4b7604-kube-api-access-lsngp\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.597979 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.597850 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78521f4f-2f40-4d1f-8490-d0764d4b7604-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.597979 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.597868 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.597979 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.597881 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78521f4f-2f40-4d1f-8490-d0764d4b7604-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.623288 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.623256 4934 generic.go:356] "Generic (PLEG): container finished" podID="78521f4f-2f40-4d1f-8490-d0764d4b7604" containerID="80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c" exitCode=0 Oct 31 04:11:46.623427 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.623298 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" event={"ID":"78521f4f-2f40-4d1f-8490-d0764d4b7604","Type":"ContainerDied","Data":"80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c"} Oct 31 04:11:46.623427 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.623330 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" event={"ID":"78521f4f-2f40-4d1f-8490-d0764d4b7604","Type":"ContainerDied","Data":"66f0fbf5bccf30b1b337ff5bc8acac3e7f44c092a271ce44f0604fc6d81b9a59"} Oct 31 04:11:46.623427 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.623346 4934 scope.go:122] "RemoveContainer" containerID="80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c" Oct 31 04:11:46.623427 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.623367 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" Oct 31 04:11:46.632806 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.632783 4934 scope.go:122] "RemoveContainer" containerID="80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c" Oct 31 04:11:46.633124 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:11:46.633096 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c\": container with ID starting with 80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c not found: ID does not exist" containerID="80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c" Oct 31 04:11:46.633270 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.633136 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c"} err="failed to get container status \"80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c\": rpc error: code = NotFound desc = could not find container \"80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c\": container with ID starting with 80a573b3fb2b2a303bf3337b05119d85bbf21a90670c2c094a288a063def2d8c not found: ID does not exist" Oct 31 04:11:46.786796 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.786717 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l"] Oct 31 04:11:46.791405 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.791379 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l"] Oct 31 04:11:46.821723 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:46.821689 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78521f4f-2f40-4d1f-8490-d0764d4b7604" path="/var/lib/kubelet/pods/78521f4f-2f40-4d1f-8490-d0764d4b7604/volumes" Oct 31 04:11:47.337886 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:47.337842 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-20cd9e/simulator-single-main-20cd9e-kserve-68d5c49557-jr42l" podUID="78521f4f-2f40-4d1f-8490-d0764d4b7604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 04:11:49.165593 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:11:49.165558 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:11:53.759539 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:53.759500 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4"] Oct 31 04:11:53.760129 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:53.759690 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" containerID="cri-o://ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f" gracePeriod=30 Oct 31 04:11:59.011312 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.011288 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:11:59.104394 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104368 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0278ce-8527-4f03-9973-7838ecdf03b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0278ce-8527-4f03-9973-7838ecdf03b2-istiod-ca-cert\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104532 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104409 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-envoy\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104532 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104435 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-data\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104654 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104539 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-kube-api-access-k52qs\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-kube-api-access-k52qs\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104654 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104624 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-token\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-token\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104820 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104671 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-podinfo\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104820 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104699 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-data" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:59.104820 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104700 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-certs\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104820 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104746 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-socket\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.104820 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104759 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0278ce-8527-4f03-9973-7838ecdf03b2-istiod-ca-cert" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:59.104820 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104783 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-credential-socket\") pod \"1c0278ce-8527-4f03-9973-7838ecdf03b2\" (UID: \"1c0278ce-8527-4f03-9973-7838ecdf03b2\") " Oct 31 04:11:59.105142 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.104916 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-certs" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:59.105142 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.105045 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-socket" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:59.105142 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.105091 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.105142 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.105113 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0278ce-8527-4f03-9973-7838ecdf03b2-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.105142 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.105129 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.105142 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.105136 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-credential-socket" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:59.106827 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.106803 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-envoy" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:59.106918 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.106854 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-kube-api-access-k52qs" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "kube-api-access-k52qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:59.106969 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.106926 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-token" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:59.107203 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.107179 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-podinfo" pod "1c0278ce-8527-4f03-9973-7838ecdf03b2" (UID: "1c0278ce-8527-4f03-9973-7838ecdf03b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:11:59.206243 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.206209 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.206379 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.206249 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.206379 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.206260 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.206379 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.206269 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.206379 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.206277 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0278ce-8527-4f03-9973-7838ecdf03b2-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.206379 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.206285 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52qs\" (UniqueName: \"kubernetes.io/projected/1c0278ce-8527-4f03-9973-7838ecdf03b2-kube-api-access-k52qs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.689055 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.689021 4934 generic.go:356] "Generic (PLEG): container finished" podID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerID="ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f" exitCode=0 Oct 31 04:11:59.689252 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.689065 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" Oct 31 04:11:59.689252 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.689100 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" event={"ID":"1c0278ce-8527-4f03-9973-7838ecdf03b2","Type":"ContainerDied","Data":"ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f"} Oct 31 04:11:59.689252 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.689149 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4" event={"ID":"1c0278ce-8527-4f03-9973-7838ecdf03b2","Type":"ContainerDied","Data":"7332f3ff25738a151b71d713b1efb2d34cc66966e3692bfae28780682ad8eb0a"} Oct 31 04:11:59.689252 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.689171 4934 scope.go:122] "RemoveContainer" containerID="ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f" Oct 31 04:11:59.698195 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.698178 4934 scope.go:122] "RemoveContainer" containerID="4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0" Oct 31 04:11:59.706021 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.706003 4934 scope.go:122] "RemoveContainer" containerID="ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f" Oct 31 04:11:59.706291 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:11:59.706273 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f\": container with ID starting with ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f not found: ID does not exist" containerID="ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f" Oct 31 04:11:59.706352 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.706300 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f"} err="failed to get container status \"ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f\": rpc error: code = NotFound desc = could not find container \"ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f\": container with ID starting with ca49f0c11141a5ffb2d6ec65ffde2e73e1e4a8732c62fccd4a3471a022eae13f not found: ID does not exist" Oct 31 04:11:59.706352 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.706316 4934 scope.go:122] "RemoveContainer" containerID="4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0" Oct 31 04:11:59.706545 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:11:59.706524 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0\": container with ID starting with 4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0 not found: ID does not exist" containerID="4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0" Oct 31 04:11:59.706605 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.706556 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0"} err="failed to get container status \"4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0\": rpc error: code = NotFound desc = could not find container \"4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0\": container with ID starting with 4911928fbaa2b9924023a7825d30154fbb653949d979127bf18c19121cc4e9e0 not found: ID does not exist" Oct 31 04:11:59.853933 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.853896 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4"] Oct 31 04:11:59.903925 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:11:59.903898 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-20cd9e-openshift-default-8565b9cc4ffxws4"] Oct 31 04:12:00.819975 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:00.819944 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" path="/var/lib/kubelet/pods/1c0278ce-8527-4f03-9973-7838ecdf03b2/volumes" Oct 31 04:12:19.164173 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:12:19.164127 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:12:33.341035 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.340996 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4"] Oct 31 04:12:33.341631 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.341359 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" podUID="e2318925-8599-4d5c-9ba7-ca6626bfa997" containerName="manager" containerID="cri-o://f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1" gracePeriod=10 Oct 31 04:12:33.605566 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.605486 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 04:12:33.734949 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.734913 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2318925-8599-4d5c-9ba7-ca6626bfa997-kube-api-access-4p28m\" (UniqueName: \"kubernetes.io/projected/e2318925-8599-4d5c-9ba7-ca6626bfa997-kube-api-access-4p28m\") pod \"e2318925-8599-4d5c-9ba7-ca6626bfa997\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " Oct 31 04:12:33.735142 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.735081 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2318925-8599-4d5c-9ba7-ca6626bfa997-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2318925-8599-4d5c-9ba7-ca6626bfa997-extensions-socket-volume\") pod \"e2318925-8599-4d5c-9ba7-ca6626bfa997\" (UID: \"e2318925-8599-4d5c-9ba7-ca6626bfa997\") " Oct 31 04:12:33.736268 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.735853 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2318925-8599-4d5c-9ba7-ca6626bfa997-extensions-socket-volume" pod "e2318925-8599-4d5c-9ba7-ca6626bfa997" (UID: "e2318925-8599-4d5c-9ba7-ca6626bfa997"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:33.738625 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.738476 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2318925-8599-4d5c-9ba7-ca6626bfa997-kube-api-access-4p28m" pod "e2318925-8599-4d5c-9ba7-ca6626bfa997" (UID: "e2318925-8599-4d5c-9ba7-ca6626bfa997"). InnerVolumeSpecName "kube-api-access-4p28m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:33.836425 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.836390 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p28m\" (UniqueName: \"kubernetes.io/projected/e2318925-8599-4d5c-9ba7-ca6626bfa997-kube-api-access-4p28m\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:12:33.836425 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.836424 4934 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2318925-8599-4d5c-9ba7-ca6626bfa997-extensions-socket-volume\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:12:33.882607 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.882512 4934 generic.go:356] "Generic (PLEG): container finished" podID="e2318925-8599-4d5c-9ba7-ca6626bfa997" containerID="f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1" exitCode=0 Oct 31 04:12:33.882607 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.882573 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" Oct 31 04:12:33.882807 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.882603 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" event={"ID":"e2318925-8599-4d5c-9ba7-ca6626bfa997","Type":"ContainerDied","Data":"f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1"} Oct 31 04:12:33.882807 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.882668 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4" event={"ID":"e2318925-8599-4d5c-9ba7-ca6626bfa997","Type":"ContainerDied","Data":"119d095929088771000ba03fad93525b7e3291274fda4a33da3b7d3be85eb3b6"} Oct 31 04:12:33.882807 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.882691 4934 scope.go:122] "RemoveContainer" containerID="f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1" Oct 31 04:12:33.895199 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.895175 4934 scope.go:122] "RemoveContainer" containerID="f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1" Oct 31 04:12:33.895662 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:12:33.895641 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1\": container with ID starting with f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1 not found: ID does not exist" containerID="f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1" Oct 31 04:12:33.895714 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:33.895674 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1"} err="failed to get container status \"f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1\": rpc error: code = NotFound desc = could not find container \"f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1\": container with ID starting with f9ae7abf5a0be7d3c10c803e3e75092b40984fbf666509f21346cb5b7f9655e1 not found: ID does not exist" Oct 31 04:12:34.048080 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:34.048047 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4"] Oct 31 04:12:34.052162 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:34.052135 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k9vh4"] Oct 31 04:12:34.820503 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:12:34.820469 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2318925-8599-4d5c-9ba7-ca6626bfa997" path="/var/lib/kubelet/pods/e2318925-8599-4d5c-9ba7-ca6626bfa997/volumes" Oct 31 04:12:49.163555 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:12:49.163472 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:13:19.162651 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:13:19.162605 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:13:36.956815 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.956780 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq"] Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957135 4934 cpu_manager.go:414] "containerMap: removing container" podUID="e2318925-8599-4d5c-9ba7-ca6626bfa997" containerName="manager" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957147 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2318925-8599-4d5c-9ba7-ca6626bfa997" containerName="manager" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957164 4934 cpu_manager.go:414] "containerMap: removing container" podUID="78521f4f-2f40-4d1f-8490-d0764d4b7604" containerName="main" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957169 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78521f4f-2f40-4d1f-8490-d0764d4b7604" containerName="main" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957182 4934 cpu_manager.go:414] "containerMap: removing container" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957187 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957195 4934 cpu_manager.go:414] "containerMap: removing container" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957199 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" Oct 31 04:13:36.957276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957279 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="78521f4f-2f40-4d1f-8490-d0764d4b7604" containerName="main" Oct 31 04:13:36.957605 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957293 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" Oct 31 04:13:36.957605 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957303 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0278ce-8527-4f03-9973-7838ecdf03b2" containerName="istio-proxy" Oct 31 04:13:36.957605 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.957317 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2318925-8599-4d5c-9ba7-ca6626bfa997" containerName="manager" Oct 31 04:13:36.960363 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:36.960343 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.015754 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.015716 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq"] Oct 31 04:13:37.047929 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.047889 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqgkg\" (UniqueName: \"kubernetes.io/projected/7e751df4-44e7-4fd6-856b-34586f7a28fa-kube-api-access-mqgkg\") pod \"kuadrant-operator-controller-manager-678d87766f-nhqmq\" (UID: \"7e751df4-44e7-4fd6-856b-34586f7a28fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.048095 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.047952 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e751df4-44e7-4fd6-856b-34586f7a28fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nhqmq\" (UID: \"7e751df4-44e7-4fd6-856b-34586f7a28fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.064071 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.064036 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 04:13:37.064071 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.064056 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v67zc\"" Oct 31 04:13:37.064257 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.064107 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 04:13:37.148720 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.148682 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqgkg\" (UniqueName: \"kubernetes.io/projected/7e751df4-44e7-4fd6-856b-34586f7a28fa-kube-api-access-mqgkg\") pod \"kuadrant-operator-controller-manager-678d87766f-nhqmq\" (UID: \"7e751df4-44e7-4fd6-856b-34586f7a28fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.148909 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.148736 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e751df4-44e7-4fd6-856b-34586f7a28fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nhqmq\" (UID: \"7e751df4-44e7-4fd6-856b-34586f7a28fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.149103 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.149084 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e751df4-44e7-4fd6-856b-34586f7a28fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nhqmq\" (UID: \"7e751df4-44e7-4fd6-856b-34586f7a28fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.210416 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.210341 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqgkg\" (UniqueName: \"kubernetes.io/projected/7e751df4-44e7-4fd6-856b-34586f7a28fa-kube-api-access-mqgkg\") pod \"kuadrant-operator-controller-manager-678d87766f-nhqmq\" (UID: \"7e751df4-44e7-4fd6-856b-34586f7a28fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.272525 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.272491 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:37.703188 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:37.703152 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq"] Oct 31 04:13:37.706817 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:13:37.706789 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e751df4_44e7_4fd6_856b_34586f7a28fa.slice/crio-3f2e39f758c710f502a8264f4d9bbcee738f2f7ce76a850c5b8d21acf717ee9a WatchSource:0}: Error finding container 3f2e39f758c710f502a8264f4d9bbcee738f2f7ce76a850c5b8d21acf717ee9a: Status 404 returned error can't find the container with id 3f2e39f758c710f502a8264f4d9bbcee738f2f7ce76a850c5b8d21acf717ee9a Oct 31 04:13:38.161497 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:38.161463 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" event={"ID":"7e751df4-44e7-4fd6-856b-34586f7a28fa","Type":"ContainerStarted","Data":"a2e05f4748054c03b226a16eaa8b689b1db245a1f10cf326da5efac81f69024e"} Oct 31 04:13:38.161886 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:38.161521 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" event={"ID":"7e751df4-44e7-4fd6-856b-34586f7a28fa","Type":"ContainerStarted","Data":"3f2e39f758c710f502a8264f4d9bbcee738f2f7ce76a850c5b8d21acf717ee9a"} Oct 31 04:13:38.161886 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:38.161564 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:13:38.298604 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:38.298538 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" podStartSLOduration=2.298523492 podStartE2EDuration="2.298523492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:38.29724797 +0000 UTC m=+3251.615990587" watchObservedRunningTime="2026-10-31 04:13:38.298523492 +0000 UTC m=+3251.617266185" Oct 31 04:13:49.161131 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:13:49.161097 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:13:49.168126 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:13:49.168103 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nhqmq" Oct 31 04:14:19.162501 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:14:19.162462 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:14:24.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:24.361060 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-q9mpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:14:24.363256 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:24.361131 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-q9mpw" podUID="b9444d14-3d85-439f-a7ef-751d9fee021e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:14:27.751275 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:27.751203 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:14:27.815377 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:27.815349 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:14:46.233824 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.233780 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4"] Oct 31 04:14:46.238603 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.238578 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.291055 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.291023 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-09c573-openshift-default-dockercfg-w5ft7\"" Oct 31 04:14:46.343089 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.343059 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4"] Oct 31 04:14:46.373985 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.373949 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-credential-socket\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374189 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374057 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04991848-498d-4451-830f-f97879ea5f09-istiod-ca-cert\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374189 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374089 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-socket\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374189 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374140 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-envoy\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374413 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374192 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04991848-498d-4451-830f-f97879ea5f09-istio-podinfo\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374413 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374315 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-data\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374413 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374405 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86f4\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-kube-api-access-t86f4\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374550 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374487 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-istio-token\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.374550 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.374526 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-certs\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.475708 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475662 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04991848-498d-4451-830f-f97879ea5f09-istiod-ca-cert\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.475920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475719 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-socket\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.475920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475763 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-envoy\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.475920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475788 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04991848-498d-4451-830f-f97879ea5f09-istio-podinfo\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.475920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475816 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-data\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.475920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475850 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86f4\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-kube-api-access-t86f4\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.475920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475891 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-istio-token\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.476280 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475922 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-certs\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.476280 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.475963 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-credential-socket\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.476280 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.476188 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-socket\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.476528 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.476507 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-data\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.476909 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.476887 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-credential-socket\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.477127 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.477104 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-certs\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.477875 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.477851 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04991848-498d-4451-830f-f97879ea5f09-istiod-ca-cert\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.483277 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.480001 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-envoy\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.483277 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.480075 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04991848-498d-4451-830f-f97879ea5f09-istio-podinfo\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.537949 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.537865 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-istio-token\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.538144 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.538100 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86f4\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-kube-api-access-t86f4\") pod \"e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.552718 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.552691 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:46.986134 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.986067 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4"] Oct 31 04:14:46.988363 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:14:46.988333 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04991848_498d_4451_830f_f97879ea5f09.slice/crio-81956f2fc4bffcee0ae413b75843c05ee1611d3d0d9d8d64881509f3e0b22da6 WatchSource:0}: Error finding container 81956f2fc4bffcee0ae413b75843c05ee1611d3d0d9d8d64881509f3e0b22da6: Status 404 returned error can't find the container with id 81956f2fc4bffcee0ae413b75843c05ee1611d3d0d9d8d64881509f3e0b22da6 Oct 31 04:14:46.990563 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.990535 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:14:46.990656 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.990601 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:14:46.990656 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:46.990630 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:14:47.512303 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:47.512269 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" event={"ID":"04991848-498d-4451-830f-f97879ea5f09","Type":"ContainerStarted","Data":"df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c"} Oct 31 04:14:47.512681 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:47.512323 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" event={"ID":"04991848-498d-4451-830f-f97879ea5f09","Type":"ContainerStarted","Data":"81956f2fc4bffcee0ae413b75843c05ee1611d3d0d9d8d64881509f3e0b22da6"} Oct 31 04:14:47.552962 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:47.552938 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:47.636450 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:47.636395 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" podStartSLOduration=1.636378739 podStartE2EDuration="1.636378739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:47.634346036 +0000 UTC m=+3320.953088667" watchObservedRunningTime="2026-10-31 04:14:47.636378739 +0000 UTC m=+3320.955121369" Oct 31 04:14:48.558526 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:48.558496 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:49.162802 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:14:49.162771 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:14:49.523936 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:49.523841 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:14:49.524873 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:14:49.524852 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:15:14.630950 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:14.630915 4934 generic.go:356] "Generic (PLEG): container finished" podID="04991848-498d-4451-830f-f97879ea5f09" containerID="df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c" exitCode=0 Oct 31 04:15:14.631384 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:14.630985 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" event={"ID":"04991848-498d-4451-830f-f97879ea5f09","Type":"ContainerDied","Data":"df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c"} Oct 31 04:15:14.631384 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:14.631335 4934 scope.go:122] "RemoveContainer" containerID="df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c" Oct 31 04:15:14.632367 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:14.632343 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:15:14.632433 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:14.632419 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:15:14.632470 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:14.632459 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:15:15.637453 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:15.637421 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" event={"ID":"04991848-498d-4451-830f-f97879ea5f09","Type":"ContainerStarted","Data":"d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57"} Oct 31 04:15:16.572252 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:16.562517 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:15:16.578254 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:16.577325 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 04:15:16.578254 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:16.577377 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 04:15:17.558997 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:17.558573 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 04:15:17.558997 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:17.558630 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 04:15:18.553021 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:18.552985 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 04:15:18.553209 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:18.553044 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 04:15:19.188739 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:15:19.188669 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:15:19.523814 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:19.523729 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:15:19.557255 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:19.553642 4934 patch_prober.go:29] interesting pod/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 04:15:19.557255 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:19.553715 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 04:15:20.557570 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.557538 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:15:20.558590 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.558573 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:15:20.661025 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.660999 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp"] Oct 31 04:15:20.663341 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.663318 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.691854 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.691821 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp"] Oct 31 04:15:20.715894 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.715842 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-09c573\"/\"kube-root-ca.crt\"" Oct 31 04:15:20.716044 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.715993 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-09c573\"/\"default-dockercfg-4wv75\"" Oct 31 04:15:20.716044 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.716012 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-09c573\"/\"openshift-service-ca.crt\"" Oct 31 04:15:20.716044 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.716015 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-09c573\"/\"simulator-single-main-09c573-kserve-self-signed-certs\"" Oct 31 04:15:20.725199 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.725176 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7czh\" (UniqueName: \"kubernetes.io/projected/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-kube-api-access-h7czh\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.725326 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.725242 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-dshm\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.725403 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.725390 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-home\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.725445 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.725429 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tmp-dir\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.725486 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.725456 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tls-certs\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.725550 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.725533 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-model-cache\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826389 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826305 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7czh\" (UniqueName: \"kubernetes.io/projected/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-kube-api-access-h7czh\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826389 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826345 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-dshm\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826389 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826372 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-home\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826683 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826401 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tmp-dir\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826683 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826426 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tls-certs\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826683 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826443 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-model-cache\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826850 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826826 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tmp-dir\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826894 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826851 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-home\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.826894 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.826886 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-model-cache\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.828777 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.828757 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-dshm\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.829032 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.829014 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tls-certs\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.888010 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.887975 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7czh\" (UniqueName: \"kubernetes.io/projected/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-kube-api-access-h7czh\") pod \"simulator-single-main-09c573-kserve-7474c69b9f-s6vmp\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:20.980610 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:20.980575 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:21.353154 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:21.353131 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp"] Oct 31 04:15:21.355450 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:15:21.355418 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7319ad_a4cf_44e7_a8a2_7a074b13a0b6.slice/crio-109e20d9ea885bb0aa7fabd12f2e7d394dbd4a8bdc22e300ea6dafa8cdcf159b WatchSource:0}: Error finding container 109e20d9ea885bb0aa7fabd12f2e7d394dbd4a8bdc22e300ea6dafa8cdcf159b: Status 404 returned error can't find the container with id 109e20d9ea885bb0aa7fabd12f2e7d394dbd4a8bdc22e300ea6dafa8cdcf159b Oct 31 04:15:21.683004 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:21.682917 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" event={"ID":"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6","Type":"ContainerStarted","Data":"23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3"} Oct 31 04:15:21.683004 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:21.682964 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" event={"ID":"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6","Type":"ContainerStarted","Data":"109e20d9ea885bb0aa7fabd12f2e7d394dbd4a8bdc22e300ea6dafa8cdcf159b"} Oct 31 04:15:21.800776 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:21.800726 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" podStartSLOduration=1.800713305 podStartE2EDuration="1.800713305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:21.79957248 +0000 UTC m=+3355.118315109" watchObservedRunningTime="2026-10-31 04:15:21.800713305 +0000 UTC m=+3355.119455928" Oct 31 04:15:21.981341 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:21.981265 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:30.981459 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:30.981410 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:30.993932 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:30.993904 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:31.741322 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:15:31.741291 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:15:49.163461 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:15:49.163382 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:16:19.162661 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:16:19.162627 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:16:20.622020 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:20.621982 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp"] Oct 31 04:16:35.642183 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:35.642119 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" podUID="ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" containerName="main" containerID="cri-o://23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3" gracePeriod=45 Oct 31 04:16:35.730489 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:35.730449 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" podUID="ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": dial tcp 10.132.0.103:8000: connect: connection refused" Oct 31 04:16:35.889320 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:35.889291 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:16:36.004295 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004167 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tmp-dir\") pod \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " Oct 31 04:16:36.004497 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004299 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-kube-api-access-h7czh\" (UniqueName: \"kubernetes.io/projected/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-kube-api-access-h7czh\") pod \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " Oct 31 04:16:36.004497 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004329 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-home\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-home\") pod \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " Oct 31 04:16:36.004497 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004441 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tmp-dir" pod "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" (UID: "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:36.004497 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004490 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-dshm\") pod \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " Oct 31 04:16:36.004734 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004552 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tls-certs\") pod \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " Oct 31 04:16:36.004734 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004591 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-model-cache\") pod \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\" (UID: \"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6\") " Oct 31 04:16:36.004734 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004599 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-home" pod "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" (UID: "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:36.004913 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004890 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-model-cache" pod "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" (UID: "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:36.004992 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004975 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:36.005038 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.004999 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:36.006977 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.006945 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-kube-api-access-h7czh" pod "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" (UID: "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6"). InnerVolumeSpecName "kube-api-access-h7czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:36.007190 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.007136 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-dshm" pod "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" (UID: "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:36.007190 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.007168 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tls-certs" pod "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" (UID: "ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:36.020967 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.020932 4934 generic.go:356] "Generic (PLEG): container finished" podID="ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" containerID="23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3" exitCode=0 Oct 31 04:16:36.021107 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.020979 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" event={"ID":"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6","Type":"ContainerDied","Data":"23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3"} Oct 31 04:16:36.021107 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.021013 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" event={"ID":"ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6","Type":"ContainerDied","Data":"109e20d9ea885bb0aa7fabd12f2e7d394dbd4a8bdc22e300ea6dafa8cdcf159b"} Oct 31 04:16:36.021107 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.021034 4934 scope.go:122] "RemoveContainer" containerID="23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3" Oct 31 04:16:36.021107 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.020984 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp" Oct 31 04:16:36.031741 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.031715 4934 scope.go:122] "RemoveContainer" containerID="23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3" Oct 31 04:16:36.032083 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:16:36.032062 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3\": container with ID starting with 23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3 not found: ID does not exist" containerID="23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3" Oct 31 04:16:36.032166 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.032093 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3"} err="failed to get container status \"23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3\": rpc error: code = NotFound desc = could not find container \"23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3\": container with ID starting with 23748262210f3198a5447f92abcdd655f0c43f699c9664b5fb3bb963583cc2a3 not found: ID does not exist" Oct 31 04:16:36.105987 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.105957 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:36.105987 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.105982 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:36.105987 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.105993 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:36.106274 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.106004 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7czh\" (UniqueName: \"kubernetes.io/projected/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6-kube-api-access-h7czh\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:36.185310 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.185271 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp"] Oct 31 04:16:36.191604 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.191576 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-09c573/simulator-single-main-09c573-kserve-7474c69b9f-s6vmp"] Oct 31 04:16:36.824467 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:36.824434 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" path="/var/lib/kubelet/pods/ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6/volumes" Oct 31 04:16:43.006183 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:43.006147 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4"] Oct 31 04:16:43.006622 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:43.006383 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" containerID="cri-o://d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57" gracePeriod=30 Oct 31 04:16:48.263670 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.263645 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:16:48.323507 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323478 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-socket\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323651 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323526 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-kube-api-access-t86f4\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-kube-api-access-t86f4\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323651 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323552 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-istio-token\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-istio-token\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323651 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323596 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04991848-498d-4451-830f-f97879ea5f09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04991848-498d-4451-830f-f97879ea5f09-istio-podinfo\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323651 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323634 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-data\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323675 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-envoy\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323700 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-certs\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323752 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-credential-socket\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.323862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323792 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-socket" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:48.323862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323852 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04991848-498d-4451-830f-f97879ea5f09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04991848-498d-4451-830f-f97879ea5f09-istiod-ca-cert\") pod \"04991848-498d-4451-830f-f97879ea5f09\" (UID: \"04991848-498d-4451-830f-f97879ea5f09\") " Oct 31 04:16:48.324126 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.323991 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-credential-socket" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:48.324183 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.324160 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.324183 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.324167 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-certs" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:48.324183 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.324178 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.324411 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.324375 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-data" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:48.324583 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.324559 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04991848-498d-4451-830f-f97879ea5f09-istiod-ca-cert" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:48.326199 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.326166 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04991848-498d-4451-830f-f97879ea5f09-istio-podinfo" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:16:48.326392 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.326222 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-kube-api-access-t86f4" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "kube-api-access-t86f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:48.326392 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.326371 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-istio-token" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:48.327345 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.327317 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-envoy" pod "04991848-498d-4451-830f-f97879ea5f09" (UID: "04991848-498d-4451-830f-f97879ea5f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:16:48.425484 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.425454 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04991848-498d-4451-830f-f97879ea5f09-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.425484 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.425481 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86f4\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-kube-api-access-t86f4\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.425647 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.425493 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04991848-498d-4451-830f-f97879ea5f09-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.425647 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.425501 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04991848-498d-4451-830f-f97879ea5f09-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.425647 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.425509 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.425647 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.425518 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:48.425647 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:48.425526 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04991848-498d-4451-830f-f97879ea5f09-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:49.082297 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.082181 4934 generic.go:356] "Generic (PLEG): container finished" podID="04991848-498d-4451-830f-f97879ea5f09" containerID="d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57" exitCode=0 Oct 31 04:16:49.082297 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.082263 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" Oct 31 04:16:49.082297 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.082276 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" event={"ID":"04991848-498d-4451-830f-f97879ea5f09","Type":"ContainerDied","Data":"d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57"} Oct 31 04:16:49.082542 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.082346 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4" event={"ID":"04991848-498d-4451-830f-f97879ea5f09","Type":"ContainerDied","Data":"81956f2fc4bffcee0ae413b75843c05ee1611d3d0d9d8d64881509f3e0b22da6"} Oct 31 04:16:49.082542 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.082378 4934 scope.go:122] "RemoveContainer" containerID="d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57" Oct 31 04:16:49.091453 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.091433 4934 scope.go:122] "RemoveContainer" containerID="df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c" Oct 31 04:16:49.099126 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.099095 4934 scope.go:122] "RemoveContainer" containerID="d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57" Oct 31 04:16:49.099468 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:16:49.099447 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57\": container with ID starting with d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57 not found: ID does not exist" containerID="d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57" Oct 31 04:16:49.099572 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.099479 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57"} err="failed to get container status \"d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57\": rpc error: code = NotFound desc = could not find container \"d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57\": container with ID starting with d450c3566c466faa8bf859f91d07fe806c1ccfcd80ee8da63acbe53ea2d5ec57 not found: ID does not exist" Oct 31 04:16:49.099572 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.099501 4934 scope.go:122] "RemoveContainer" containerID="df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c" Oct 31 04:16:49.099807 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:16:49.099782 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c\": container with ID starting with df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c not found: ID does not exist" containerID="df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c" Oct 31 04:16:49.099865 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.099815 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c"} err="failed to get container status \"df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c\": rpc error: code = NotFound desc = could not find container \"df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c\": container with ID starting with df95e6f9eee100c2cb40b18e2dc83a30ef99ea60ccddd26546bedd4e64e4fe2c not found: ID does not exist" Oct 31 04:16:49.162138 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:16:49.162088 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:16:49.296932 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.296894 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4"] Oct 31 04:16:49.346939 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:49.346866 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-09c573-openshift-default-9f6bbfd4f-4vdh4"] Oct 31 04:16:50.767436 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.767394 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh"] Oct 31 04:16:50.767931 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.767911 4934 cpu_manager.go:414] "containerMap: removing container" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" Oct 31 04:16:50.768071 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.767934 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" Oct 31 04:16:50.768071 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.767975 4934 cpu_manager.go:414] "containerMap: removing container" podUID="ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" containerName="main" Oct 31 04:16:50.768071 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.767983 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" containerName="main" Oct 31 04:16:50.768221 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.768081 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7319ad-a4cf-44e7-a8a2-7a074b13a0b6" containerName="main" Oct 31 04:16:50.768221 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.768093 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" Oct 31 04:16:50.768221 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.768106 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" Oct 31 04:16:50.772539 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.772518 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.820403 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.820371 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04991848-498d-4451-830f-f97879ea5f09" path="/var/lib/kubelet/pods/04991848-498d-4451-830f-f97879ea5f09/volumes" Oct 31 04:16:50.843781 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.843755 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzkh\" (UniqueName: \"kubernetes.io/projected/dc062588-f4af-4de2-950b-612e71f6d88e-kube-api-access-lzzkh\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.843918 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.843789 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-dshm\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.843918 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.843812 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-model-cache\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.843918 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.843850 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc062588-f4af-4de2-950b-612e71f6d88e-tls-certs\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.843918 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.843909 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-tmp-dir\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.844070 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.843935 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-home\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.875012 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.874989 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 04:16:50.875148 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.875009 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6fa8b670-kserve-self-signed-certs\"" Oct 31 04:16:50.875148 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.875012 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 04:16:50.875148 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.874993 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j8tx4\"" Oct 31 04:16:50.876375 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.876356 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh"] Oct 31 04:16:50.945268 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945212 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzkh\" (UniqueName: \"kubernetes.io/projected/dc062588-f4af-4de2-950b-612e71f6d88e-kube-api-access-lzzkh\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945268 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945272 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-dshm\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945500 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945304 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-model-cache\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945500 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945326 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc062588-f4af-4de2-950b-612e71f6d88e-tls-certs\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945500 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945346 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-tmp-dir\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945500 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945380 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-home\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945759 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945729 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-model-cache\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945759 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945746 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-home\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.945900 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.945768 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-tmp-dir\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.947644 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.947624 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-dshm\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:50.947909 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:50.947894 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc062588-f4af-4de2-950b-612e71f6d88e-tls-certs\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:51.006567 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:51.006539 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzkh\" (UniqueName: \"kubernetes.io/projected/dc062588-f4af-4de2-950b-612e71f6d88e-kube-api-access-lzzkh\") pod \"e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:51.099746 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:51.099717 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:51.481922 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:51.481897 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh"] Oct 31 04:16:51.484483 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:16:51.484447 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc062588_f4af_4de2_950b_612e71f6d88e.slice/crio-6b4f7e4146451b0351c12649c3bd6fad999e8dd1269a44d25e9a7076fa7ac111 WatchSource:0}: Error finding container 6b4f7e4146451b0351c12649c3bd6fad999e8dd1269a44d25e9a7076fa7ac111: Status 404 returned error can't find the container with id 6b4f7e4146451b0351c12649c3bd6fad999e8dd1269a44d25e9a7076fa7ac111 Oct 31 04:16:52.097409 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:52.097370 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" event={"ID":"dc062588-f4af-4de2-950b-612e71f6d88e","Type":"ContainerStarted","Data":"87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa"} Oct 31 04:16:52.097867 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:52.097416 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" event={"ID":"dc062588-f4af-4de2-950b-612e71f6d88e","Type":"ContainerStarted","Data":"6b4f7e4146451b0351c12649c3bd6fad999e8dd1269a44d25e9a7076fa7ac111"} Oct 31 04:16:52.100613 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:52.100597 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:16:52.214383 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:52.214194 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" podStartSLOduration=2.214175691 podStartE2EDuration="2.214175691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:52.213735831 +0000 UTC m=+3445.532478458" watchObservedRunningTime="2026-10-31 04:16:52.214175691 +0000 UTC m=+3445.532918321" Oct 31 04:16:57.743463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:57.743417 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s5kcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:16:57.743996 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:16:57.743477 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s5kcb" podUID="5453941c-2d09-4ff9-aaaf-cc0e162876e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:01.100613 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:01.100576 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:17:01.113785 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:01.113761 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:17:01.160269 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:01.160243 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:17:19.162906 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:17:19.162866 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:17:34.042790 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.042748 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t"] Oct 31 04:17:34.043330 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.043145 4934 cpu_manager.go:414] "containerMap: removing container" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" Oct 31 04:17:34.043330 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.043157 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04991848-498d-4451-830f-f97879ea5f09" containerName="istio-proxy" Oct 31 04:17:34.047528 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.047503 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.148918 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.148880 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-dshm\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.148918 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.148914 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tls-certs\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.149153 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.149007 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tmp-dir\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.149153 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.149076 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-model-cache\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.149292 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.149192 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-home\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.149292 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.149254 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbklj\" (UniqueName: \"kubernetes.io/projected/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-kube-api-access-fbklj\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.149893 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.149857 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6fa8b670-kserve-self-signed-certs\"" Oct 31 04:17:34.151419 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.151401 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t"] Oct 31 04:17:34.250047 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250005 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-model-cache\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250047 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250059 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-home\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250199 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbklj\" (UniqueName: \"kubernetes.io/projected/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-kube-api-access-fbklj\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250294 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-dshm\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250323 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tls-certs\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250379 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tmp-dir\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250620 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250496 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-home\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250620 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250545 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-model-cache\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.250735 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.250699 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tmp-dir\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.253686 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.253659 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-dshm\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.255080 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.255059 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tls-certs\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.311666 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.311632 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbklj\" (UniqueName: \"kubernetes.io/projected/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-kube-api-access-fbklj\") pod \"e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.359621 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.359585 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:34.748310 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:34.748203 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t"] Oct 31 04:17:34.751287 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:17:34.751216 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d772ed9_e1e4_455c_b9e1_ff800ddbd871.slice/crio-f52842c76c0cdf6eb71c0aa9b0c6cc5be9298292797323867c18be648e009fe4 WatchSource:0}: Error finding container f52842c76c0cdf6eb71c0aa9b0c6cc5be9298292797323867c18be648e009fe4: Status 404 returned error can't find the container with id f52842c76c0cdf6eb71c0aa9b0c6cc5be9298292797323867c18be648e009fe4 Oct 31 04:17:35.300400 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:35.300360 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" event={"ID":"1d772ed9-e1e4-455c-b9e1-ff800ddbd871","Type":"ContainerStarted","Data":"7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7"} Oct 31 04:17:35.300808 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:35.300410 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" event={"ID":"1d772ed9-e1e4-455c-b9e1-ff800ddbd871","Type":"ContainerStarted","Data":"f52842c76c0cdf6eb71c0aa9b0c6cc5be9298292797323867c18be648e009fe4"} Oct 31 04:17:35.359932 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:35.359892 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:35.418333 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:35.418282 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" podStartSLOduration=1.418266037 podStartE2EDuration="1.418266037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:35.416578974 +0000 UTC m=+3488.735321604" watchObservedRunningTime="2026-10-31 04:17:35.418266037 +0000 UTC m=+3488.737008665" Oct 31 04:17:42.443471 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:42.443418 4934 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kll7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:17:42.443934 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:42.443510 4934 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kll7s" podUID="a2e58daa-d4a4-45a5-b9ae-d7906f70c916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:44.360202 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:44.360171 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:44.372504 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:44.372482 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:45.370935 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:17:45.370898 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:17:49.162803 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:17:49.162765 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:18:19.166761 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:18:19.166718 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:18:20.171657 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:20.171615 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh"] Oct 31 04:18:32.736501 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:32.736468 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t"] Oct 31 04:18:35.192326 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.192282 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" podUID="dc062588-f4af-4de2-950b-612e71f6d88e" containerName="main" containerID="cri-o://87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa" gracePeriod=45 Oct 31 04:18:35.441013 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.440985 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:18:35.492658 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.492570 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-dshm\") pod \"dc062588-f4af-4de2-950b-612e71f6d88e\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " Oct 31 04:18:35.492832 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.492665 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-model-cache\") pod \"dc062588-f4af-4de2-950b-612e71f6d88e\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " Oct 31 04:18:35.492832 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.492710 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc062588-f4af-4de2-950b-612e71f6d88e-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc062588-f4af-4de2-950b-612e71f6d88e-tls-certs\") pod \"dc062588-f4af-4de2-950b-612e71f6d88e\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " Oct 31 04:18:35.492832 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.492743 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc062588-f4af-4de2-950b-612e71f6d88e-kube-api-access-lzzkh\" (UniqueName: \"kubernetes.io/projected/dc062588-f4af-4de2-950b-612e71f6d88e-kube-api-access-lzzkh\") pod \"dc062588-f4af-4de2-950b-612e71f6d88e\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " Oct 31 04:18:35.492832 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.492776 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-tmp-dir\") pod \"dc062588-f4af-4de2-950b-612e71f6d88e\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " Oct 31 04:18:35.492832 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.492813 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-home\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-home\") pod \"dc062588-f4af-4de2-950b-612e71f6d88e\" (UID: \"dc062588-f4af-4de2-950b-612e71f6d88e\") " Oct 31 04:18:35.493086 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.492943 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-model-cache" pod "dc062588-f4af-4de2-950b-612e71f6d88e" (UID: "dc062588-f4af-4de2-950b-612e71f6d88e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:35.493086 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.493041 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-tmp-dir" pod "dc062588-f4af-4de2-950b-612e71f6d88e" (UID: "dc062588-f4af-4de2-950b-612e71f6d88e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:35.493208 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.493125 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-home" pod "dc062588-f4af-4de2-950b-612e71f6d88e" (UID: "dc062588-f4af-4de2-950b-612e71f6d88e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:35.493473 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.493450 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:35.493610 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.493476 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:35.493610 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.493490 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:35.495095 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.495067 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc062588-f4af-4de2-950b-612e71f6d88e-tls-certs" pod "dc062588-f4af-4de2-950b-612e71f6d88e" (UID: "dc062588-f4af-4de2-950b-612e71f6d88e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:35.495183 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.495115 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-dshm" pod "dc062588-f4af-4de2-950b-612e71f6d88e" (UID: "dc062588-f4af-4de2-950b-612e71f6d88e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:35.495183 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.495117 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc062588-f4af-4de2-950b-612e71f6d88e-kube-api-access-lzzkh" pod "dc062588-f4af-4de2-950b-612e71f6d88e" (UID: "dc062588-f4af-4de2-950b-612e71f6d88e"). InnerVolumeSpecName "kube-api-access-lzzkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:35.594749 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.594713 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc062588-f4af-4de2-950b-612e71f6d88e-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:35.594749 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.594741 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc062588-f4af-4de2-950b-612e71f6d88e-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:35.594749 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.594751 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzkh\" (UniqueName: \"kubernetes.io/projected/dc062588-f4af-4de2-950b-612e71f6d88e-kube-api-access-lzzkh\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:35.600578 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.600549 4934 generic.go:356] "Generic (PLEG): container finished" podID="dc062588-f4af-4de2-950b-612e71f6d88e" containerID="87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa" exitCode=0 Oct 31 04:18:35.600752 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.600584 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" event={"ID":"dc062588-f4af-4de2-950b-612e71f6d88e","Type":"ContainerDied","Data":"87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa"} Oct 31 04:18:35.600752 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.600596 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" Oct 31 04:18:35.600752 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.600611 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh" event={"ID":"dc062588-f4af-4de2-950b-612e71f6d88e","Type":"ContainerDied","Data":"6b4f7e4146451b0351c12649c3bd6fad999e8dd1269a44d25e9a7076fa7ac111"} Oct 31 04:18:35.600752 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.600634 4934 scope.go:122] "RemoveContainer" containerID="87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa" Oct 31 04:18:35.610352 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.610331 4934 scope.go:122] "RemoveContainer" containerID="87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa" Oct 31 04:18:35.610667 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:18:35.610647 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa\": container with ID starting with 87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa not found: ID does not exist" containerID="87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa" Oct 31 04:18:35.610714 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.610677 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa"} err="failed to get container status \"87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa\": rpc error: code = NotFound desc = could not find container \"87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa\": container with ID starting with 87d2007190ed2d791cfd87a3dd70850cdad32f050aa9df787804866c25bc6caa not found: ID does not exist" Oct 31 04:18:35.764717 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.764629 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh"] Oct 31 04:18:35.768420 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:35.768395 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6fa8b670-kserve-f6d947985-cgcrh"] Oct 31 04:18:36.821134 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:36.821059 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc062588-f4af-4de2-950b-612e71f6d88e" path="/var/lib/kubelet/pods/dc062588-f4af-4de2-950b-612e71f6d88e/volumes" Oct 31 04:18:47.756248 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:47.756171 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" podUID="1d772ed9-e1e4-455c-b9e1-ff800ddbd871" containerName="main" containerID="cri-o://7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7" gracePeriod=45 Oct 31 04:18:48.011455 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.011396 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:18:48.098292 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098262 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tls-certs\") pod \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " Oct 31 04:18:48.098440 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098304 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-dshm\") pod \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " Oct 31 04:18:48.098440 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098411 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-home\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-home\") pod \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " Oct 31 04:18:48.098563 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098450 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-kube-api-access-fbklj\" (UniqueName: \"kubernetes.io/projected/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-kube-api-access-fbklj\") pod \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " Oct 31 04:18:48.098563 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098505 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tmp-dir\") pod \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " Oct 31 04:18:48.098563 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098555 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-model-cache\") pod \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\" (UID: \"1d772ed9-e1e4-455c-b9e1-ff800ddbd871\") " Oct 31 04:18:48.098745 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098715 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-home" pod "1d772ed9-e1e4-455c-b9e1-ff800ddbd871" (UID: "1d772ed9-e1e4-455c-b9e1-ff800ddbd871"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:48.098854 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098837 4934 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-home\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:48.099001 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098971 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tmp-dir" pod "1d772ed9-e1e4-455c-b9e1-ff800ddbd871" (UID: "1d772ed9-e1e4-455c-b9e1-ff800ddbd871"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:48.099088 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.098981 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-model-cache" pod "1d772ed9-e1e4-455c-b9e1-ff800ddbd871" (UID: "1d772ed9-e1e4-455c-b9e1-ff800ddbd871"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:48.100943 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.100915 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-kube-api-access-fbklj" pod "1d772ed9-e1e4-455c-b9e1-ff800ddbd871" (UID: "1d772ed9-e1e4-455c-b9e1-ff800ddbd871"). InnerVolumeSpecName "kube-api-access-fbklj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:48.101156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.101132 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-dshm" pod "1d772ed9-e1e4-455c-b9e1-ff800ddbd871" (UID: "1d772ed9-e1e4-455c-b9e1-ff800ddbd871"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:18:48.101156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.101144 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tls-certs" pod "1d772ed9-e1e4-455c-b9e1-ff800ddbd871" (UID: "1d772ed9-e1e4-455c-b9e1-ff800ddbd871"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:48.199481 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.199456 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbklj\" (UniqueName: \"kubernetes.io/projected/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-kube-api-access-fbklj\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:48.199481 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.199480 4934 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tmp-dir\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:48.199626 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.199490 4934 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-model-cache\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:48.199626 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.199498 4934 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-tls-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:48.199626 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.199506 4934 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d772ed9-e1e4-455c-b9e1-ff800ddbd871-dshm\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:48.670336 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.670303 4934 generic.go:356] "Generic (PLEG): container finished" podID="1d772ed9-e1e4-455c-b9e1-ff800ddbd871" containerID="7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7" exitCode=0 Oct 31 04:18:48.670510 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.670350 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" event={"ID":"1d772ed9-e1e4-455c-b9e1-ff800ddbd871","Type":"ContainerDied","Data":"7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7"} Oct 31 04:18:48.670510 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.670408 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" event={"ID":"1d772ed9-e1e4-455c-b9e1-ff800ddbd871","Type":"ContainerDied","Data":"f52842c76c0cdf6eb71c0aa9b0c6cc5be9298292797323867c18be648e009fe4"} Oct 31 04:18:48.670510 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.670434 4934 scope.go:122] "RemoveContainer" containerID="7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7" Oct 31 04:18:48.670510 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.670361 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t" Oct 31 04:18:48.679763 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.679743 4934 scope.go:122] "RemoveContainer" containerID="7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7" Oct 31 04:18:48.680068 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:18:48.680048 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7\": container with ID starting with 7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7 not found: ID does not exist" containerID="7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7" Oct 31 04:18:48.680166 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.680077 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7"} err="failed to get container status \"7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7\": rpc error: code = NotFound desc = could not find container \"7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7\": container with ID starting with 7c6830edc13d2cf5c240d2cb8482c13af3956a13c2acc47a8bd623d733eeb0f7 not found: ID does not exist" Oct 31 04:18:48.834786 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.834755 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t"] Oct 31 04:18:48.884708 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:48.884684 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6fa8b670-kserve-785664b87c-wmn9t"] Oct 31 04:18:49.162416 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:18:49.162381 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:18:50.826597 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:18:50.826555 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d772ed9-e1e4-455c-b9e1-ff800ddbd871" path="/var/lib/kubelet/pods/1d772ed9-e1e4-455c-b9e1-ff800ddbd871/volumes" Oct 31 04:19:19.164346 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:19:19.164307 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:19:21.425878 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.425846 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd"] Oct 31 04:19:21.426463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.426223 4934 cpu_manager.go:414] "containerMap: removing container" podUID="dc062588-f4af-4de2-950b-612e71f6d88e" containerName="main" Oct 31 04:19:21.426463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.426282 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc062588-f4af-4de2-950b-612e71f6d88e" containerName="main" Oct 31 04:19:21.426463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.426305 4934 cpu_manager.go:414] "containerMap: removing container" podUID="1d772ed9-e1e4-455c-b9e1-ff800ddbd871" containerName="main" Oct 31 04:19:21.426463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.426311 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d772ed9-e1e4-455c-b9e1-ff800ddbd871" containerName="main" Oct 31 04:19:21.426463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.426378 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d772ed9-e1e4-455c-b9e1-ff800ddbd871" containerName="main" Oct 31 04:19:21.426463 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.426393 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc062588-f4af-4de2-950b-612e71f6d88e" containerName="main" Oct 31 04:19:21.429242 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.429212 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.533294 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.533257 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd"] Oct 31 04:19:21.533476 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.533260 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqvl\" (UniqueName: \"kubernetes.io/projected/260917c8-b794-455a-b94e-37f2956f8b03-kube-api-access-prqvl\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.533476 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.533383 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260917c8-b794-455a-b94e-37f2956f8b03-metrics-certs\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.533476 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.533416 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260917c8-b794-455a-b94e-37f2956f8b03-openshift-service-ca\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.533592 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.533507 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260917c8-b794-455a-b94e-37f2956f8b03-maas-api-tls\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.634946 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.634898 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqvl\" (UniqueName: \"kubernetes.io/projected/260917c8-b794-455a-b94e-37f2956f8b03-kube-api-access-prqvl\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.635172 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.634957 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260917c8-b794-455a-b94e-37f2956f8b03-metrics-certs\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.635172 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.634994 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260917c8-b794-455a-b94e-37f2956f8b03-openshift-service-ca\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.635172 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.635045 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260917c8-b794-455a-b94e-37f2956f8b03-maas-api-tls\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.635690 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.635667 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260917c8-b794-455a-b94e-37f2956f8b03-openshift-service-ca\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.637831 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.637808 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260917c8-b794-455a-b94e-37f2956f8b03-maas-api-tls\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.637949 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.637926 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260917c8-b794-455a-b94e-37f2956f8b03-metrics-certs\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.696767 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.696688 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqvl\" (UniqueName: \"kubernetes.io/projected/260917c8-b794-455a-b94e-37f2956f8b03-kube-api-access-prqvl\") pod \"maas-api-796fbdf5b4-xndbd\" (UID: \"260917c8-b794-455a-b94e-37f2956f8b03\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:21.743580 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:21.743534 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:22.117740 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:22.117708 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd"] Oct 31 04:19:22.120727 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:19:22.120695 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260917c8_b794_455a_b94e_37f2956f8b03.slice/crio-df861d3f1d2dae3f8f113082d28fc59b6438ac046fd8041cae83e2c43c53b78d WatchSource:0}: Error finding container df861d3f1d2dae3f8f113082d28fc59b6438ac046fd8041cae83e2c43c53b78d: Status 404 returned error can't find the container with id df861d3f1d2dae3f8f113082d28fc59b6438ac046fd8041cae83e2c43c53b78d Oct 31 04:19:22.122031 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:22.122010 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:19:22.841390 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:22.841351 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" event={"ID":"260917c8-b794-455a-b94e-37f2956f8b03","Type":"ContainerStarted","Data":"df861d3f1d2dae3f8f113082d28fc59b6438ac046fd8041cae83e2c43c53b78d"} Oct 31 04:19:23.847399 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:23.847363 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" event={"ID":"260917c8-b794-455a-b94e-37f2956f8b03","Type":"ContainerStarted","Data":"589f38585333fbb9ca14c178ebfce9a08f8b5caac755ef902fc5005ac3459ec5"} Oct 31 04:19:23.847903 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:23.847501 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:23.966280 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:23.966194 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" podStartSLOduration=2.10051402 podStartE2EDuration="2.966181727s" totalImagesPullingTime="865.667707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:23.965540146 +0000 UTC m=+3597.284282774" watchObservedRunningTime="2026-10-31 04:19:23.966181727 +0000 UTC m=+3597.284924354" Oct 31 04:19:27.780479 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:27.780450 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:19:27.842163 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:27.842137 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:19:29.858558 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:29.858529 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-xndbd" Oct 31 04:19:30.029504 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:30.029468 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2"] Oct 31 04:19:30.029741 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:30.029714 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" podUID="dfeebafe-bda1-4832-8c85-8564ad8febd7" containerName="maas-api" containerID="cri-o://69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178" gracePeriod=30 Oct 31 04:19:31.278001 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.277977 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 04:19:31.423518 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.423427 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-metrics-certs\") pod \"dfeebafe-bda1-4832-8c85-8564ad8febd7\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " Oct 31 04:19:31.423518 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.423469 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-maas-api-tls\") pod \"dfeebafe-bda1-4832-8c85-8564ad8febd7\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " Oct 31 04:19:31.423747 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.423562 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfeebafe-bda1-4832-8c85-8564ad8febd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeebafe-bda1-4832-8c85-8564ad8febd7-openshift-service-ca\") pod \"dfeebafe-bda1-4832-8c85-8564ad8febd7\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " Oct 31 04:19:31.423747 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.423634 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfeebafe-bda1-4832-8c85-8564ad8febd7-kube-api-access-vtskd\" (UniqueName: \"kubernetes.io/projected/dfeebafe-bda1-4832-8c85-8564ad8febd7-kube-api-access-vtskd\") pod \"dfeebafe-bda1-4832-8c85-8564ad8febd7\" (UID: \"dfeebafe-bda1-4832-8c85-8564ad8febd7\") " Oct 31 04:19:31.424018 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.423979 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfeebafe-bda1-4832-8c85-8564ad8febd7-openshift-service-ca" pod "dfeebafe-bda1-4832-8c85-8564ad8febd7" (UID: "dfeebafe-bda1-4832-8c85-8564ad8febd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:31.425841 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.425817 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-maas-api-tls" pod "dfeebafe-bda1-4832-8c85-8564ad8febd7" (UID: "dfeebafe-bda1-4832-8c85-8564ad8febd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:31.425931 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.425885 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-metrics-certs" pod "dfeebafe-bda1-4832-8c85-8564ad8febd7" (UID: "dfeebafe-bda1-4832-8c85-8564ad8febd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:31.426467 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.426452 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfeebafe-bda1-4832-8c85-8564ad8febd7-kube-api-access-vtskd" pod "dfeebafe-bda1-4832-8c85-8564ad8febd7" (UID: "dfeebafe-bda1-4832-8c85-8564ad8febd7"). InnerVolumeSpecName "kube-api-access-vtskd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:31.524645 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.524611 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtskd\" (UniqueName: \"kubernetes.io/projected/dfeebafe-bda1-4832-8c85-8564ad8febd7-kube-api-access-vtskd\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:31.524823 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.524655 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:31.524823 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.524665 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeebafe-bda1-4832-8c85-8564ad8febd7-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:31.524823 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.524674 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeebafe-bda1-4832-8c85-8564ad8febd7-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:31.883866 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.883830 4934 generic.go:356] "Generic (PLEG): container finished" podID="dfeebafe-bda1-4832-8c85-8564ad8febd7" containerID="69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178" exitCode=0 Oct 31 04:19:31.884060 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.883880 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" Oct 31 04:19:31.884060 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.883900 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" event={"ID":"dfeebafe-bda1-4832-8c85-8564ad8febd7","Type":"ContainerDied","Data":"69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178"} Oct 31 04:19:31.884060 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.883949 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2" event={"ID":"dfeebafe-bda1-4832-8c85-8564ad8febd7","Type":"ContainerDied","Data":"c7c0fe9de154dcc5cabe6e7455cd604a532c4f49ed60eeeee0cdc002f3401434"} Oct 31 04:19:31.884060 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.883970 4934 scope.go:122] "RemoveContainer" containerID="69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178" Oct 31 04:19:31.893965 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.893951 4934 scope.go:122] "RemoveContainer" containerID="69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178" Oct 31 04:19:31.894281 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:19:31.894260 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178\": container with ID starting with 69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178 not found: ID does not exist" containerID="69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178" Oct 31 04:19:31.894328 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:31.894290 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178"} err="failed to get container status \"69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178\": rpc error: code = NotFound desc = could not find container \"69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178\": container with ID starting with 69444afaa37d4c341d40ed0a3e433afbfcdafebdfa1aafaf967038cfa7468178 not found: ID does not exist" Oct 31 04:19:32.048931 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:32.048892 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2"] Oct 31 04:19:32.098693 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:32.098655 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-kslb2"] Oct 31 04:19:32.819734 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:32.819703 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfeebafe-bda1-4832-8c85-8564ad8febd7" path="/var/lib/kubelet/pods/dfeebafe-bda1-4832-8c85-8564ad8febd7/volumes" Oct 31 04:19:36.036471 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.036438 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk"] Oct 31 04:19:36.036920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.036799 4934 cpu_manager.go:414] "containerMap: removing container" podUID="dfeebafe-bda1-4832-8c85-8564ad8febd7" containerName="maas-api" Oct 31 04:19:36.036920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.036811 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfeebafe-bda1-4832-8c85-8564ad8febd7" containerName="maas-api" Oct 31 04:19:36.036920 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.036894 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfeebafe-bda1-4832-8c85-8564ad8febd7" containerName="maas-api" Oct 31 04:19:36.041015 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.040997 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.093634 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.093599 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a058534b\"" Oct 31 04:19:36.143562 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.143521 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a058534b-dockercfg-kslhc\"" Oct 31 04:19:36.144906 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.144882 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk"] Oct 31 04:19:36.166964 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.166929 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-a058534b-f69db68c-dzhnk\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.167156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.166999 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxdl\" (UniqueName: \"kubernetes.io/projected/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-kube-api-access-7nxdl\") pod \"payload-processing-e2e-ait-a058534b-f69db68c-dzhnk\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.212374 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.212328 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp"] Oct 31 04:19:36.215791 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.215766 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.267709 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.267669 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-a058534b-f69db68c-dzhnk\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.267911 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.267718 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxdl\" (UniqueName: \"kubernetes.io/projected/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-kube-api-access-7nxdl\") pod \"payload-processing-e2e-ait-a058534b-f69db68c-dzhnk\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.268544 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.268521 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-a058534b-f69db68c-dzhnk\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.274289 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.274254 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp"] Oct 31 04:19:36.329215 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.329185 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxdl\" (UniqueName: \"kubernetes.io/projected/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-kube-api-access-7nxdl\") pod \"payload-processing-e2e-ait-a058534b-f69db68c-dzhnk\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.353340 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.353305 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:36.368868 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.368836 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq5s2\" (UniqueName: \"kubernetes.io/projected/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-kube-api-access-lq5s2\") pod \"payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.369026 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.368999 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.470409 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.470368 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq5s2\" (UniqueName: \"kubernetes.io/projected/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-kube-api-access-lq5s2\") pod \"payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.470615 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.470447 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.471422 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.471399 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.532061 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.532019 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq5s2\" (UniqueName: \"kubernetes.io/projected/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-kube-api-access-lq5s2\") pod \"payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.726399 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.726312 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl"] Oct 31 04:19:36.730973 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.730954 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk"] Oct 31 04:19:36.731107 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.731095 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.732929 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:19:36.732907 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb721cfe_87d4_4afd_810b_4fbaf79ea4ab.slice/crio-a9a1f3501e9c08dd6d92e7c59107ed9d1ac685f66d1dc734ae0579c192ceba4e WatchSource:0}: Error finding container a9a1f3501e9c08dd6d92e7c59107ed9d1ac685f66d1dc734ae0579c192ceba4e: Status 404 returned error can't find the container with id a9a1f3501e9c08dd6d92e7c59107ed9d1ac685f66d1dc734ae0579c192ceba4e Oct 31 04:19:36.783523 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.783492 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a058534b-serving-cert\"" Oct 31 04:19:36.789921 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.789884 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl"] Oct 31 04:19:36.828336 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.828304 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:36.874062 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.874024 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-metrics-certs\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.874276 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.874198 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.874369 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.874349 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqtq\" (UniqueName: \"kubernetes.io/projected/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-kube-api-access-fgqtq\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.874419 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.874404 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-openshift-service-ca\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.909909 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.909880 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" event={"ID":"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab","Type":"ContainerStarted","Data":"a9a1f3501e9c08dd6d92e7c59107ed9d1ac685f66d1dc734ae0579c192ceba4e"} Oct 31 04:19:36.975425 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.975389 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.975633 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.975460 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqtq\" (UniqueName: \"kubernetes.io/projected/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-kube-api-access-fgqtq\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.975633 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.975488 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-openshift-service-ca\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.975633 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:19:36.975521 4934 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-serving-cert: secret "maas-api-e2e-ait-a058534b-serving-cert" not found Oct 31 04:19:36.975633 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:19:36.975595 4934 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls podName:a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3 nodeName:}" failed. No retries permitted until 2026-10-31 04:19:37.4755736 +0000 UTC m=+3610.794316226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls") pod "maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" (UID: "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3") : secret "maas-api-e2e-ait-a058534b-serving-cert" not found Oct 31 04:19:36.975853 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.975716 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-metrics-certs\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.976216 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.976195 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-openshift-service-ca\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:36.978735 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:36.978664 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-metrics-certs\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:37.038989 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:37.038962 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqtq\" (UniqueName: \"kubernetes.io/projected/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-kube-api-access-fgqtq\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:37.203423 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:37.203395 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp"] Oct 31 04:19:37.206023 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:19:37.205994 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b5b31b_e28b_4c1f_b46e_5b590b476d58.slice/crio-e4ce90a3556588382e8b698b619f9b892d8ed18b89bde0be23b78ed6236d08b5 WatchSource:0}: Error finding container e4ce90a3556588382e8b698b619f9b892d8ed18b89bde0be23b78ed6236d08b5: Status 404 returned error can't find the container with id e4ce90a3556588382e8b698b619f9b892d8ed18b89bde0be23b78ed6236d08b5 Oct 31 04:19:37.483087 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:37.483045 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:37.485831 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:37.485800 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls\") pod \"maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:37.651104 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:37.651050 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:37.917034 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:37.916994 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" event={"ID":"c7b5b31b-e28b-4c1f-b46e-5b590b476d58","Type":"ContainerStarted","Data":"e4ce90a3556588382e8b698b619f9b892d8ed18b89bde0be23b78ed6236d08b5"} Oct 31 04:19:38.031764 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:38.031734 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl"] Oct 31 04:19:38.033772 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:19:38.033741 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bd6a9a_c69a_4cd6_b43b_cba121ee3ac3.slice/crio-f27009410c2b4ff6e6803a68fcbf4014d5a3e52d002f5165607dd203e3d39ab9 WatchSource:0}: Error finding container f27009410c2b4ff6e6803a68fcbf4014d5a3e52d002f5165607dd203e3d39ab9: Status 404 returned error can't find the container with id f27009410c2b4ff6e6803a68fcbf4014d5a3e52d002f5165607dd203e3d39ab9 Oct 31 04:19:38.933271 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:38.928963 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" event={"ID":"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3","Type":"ContainerStarted","Data":"e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1"} Oct 31 04:19:38.933271 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:38.929028 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" event={"ID":"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3","Type":"ContainerStarted","Data":"f27009410c2b4ff6e6803a68fcbf4014d5a3e52d002f5165607dd203e3d39ab9"} Oct 31 04:19:38.933271 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:38.929072 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:39.060624 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:39.060557 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" podStartSLOduration=2.272970801 podStartE2EDuration="3.060540946s" totalImagesPullingTime="787.570145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:39.05997231 +0000 UTC m=+3612.378714940" watchObservedRunningTime="2026-10-31 04:19:39.060540946 +0000 UTC m=+3612.379283574" Oct 31 04:19:39.934396 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:39.934360 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" event={"ID":"c7b5b31b-e28b-4c1f-b46e-5b590b476d58","Type":"ContainerStarted","Data":"f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0"} Oct 31 04:19:39.934824 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:39.934531 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:39.936135 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:39.936110 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" event={"ID":"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab","Type":"ContainerStarted","Data":"5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52"} Oct 31 04:19:39.936324 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:39.936263 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:19:40.051557 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:40.051447 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" podStartSLOduration=1.435542253 podStartE2EDuration="4.051432563s" totalImagesPullingTime="2.61589031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:40.04980949 +0000 UTC m=+3613.368552120" watchObservedRunningTime="2026-10-31 04:19:40.051432563 +0000 UTC m=+3613.370175189" Oct 31 04:19:40.165807 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:40.165730 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" podStartSLOduration=2.205977959 podStartE2EDuration="5.165710852s" totalImagesPullingTime="2.959732893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:40.164998173 +0000 UTC m=+3613.483740807" watchObservedRunningTime="2026-10-31 04:19:40.165710852 +0000 UTC m=+3613.484453487" Oct 31 04:19:44.941905 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:44.941878 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:19:49.162462 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:19:49.162428 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:19:50.944197 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:50.944160 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:19:50.944646 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:19:50.944241 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:20:19.162681 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:20:19.162645 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:20:20.418245 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:20.418200 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk"] Oct 31 04:20:20.528601 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:20.528566 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp"] Oct 31 04:20:25.418766 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.418727 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" podUID="eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" containerName="payload-processing" containerID="cri-o://5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52" gracePeriod=25 Oct 31 04:20:25.496689 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.496665 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl"] Oct 31 04:20:25.496904 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.496881 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" podUID="a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" containerName="maas-api" containerID="cri-o://e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1" gracePeriod=30 Oct 31 04:20:25.529707 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.529677 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" podUID="c7b5b31b-e28b-4c1f-b46e-5b590b476d58" containerName="payload-pre-processing" containerID="cri-o://f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0" gracePeriod=25 Oct 31 04:20:25.790434 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.790408 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:20:25.812734 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.812712 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:20:25.822879 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.822858 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:20:25.886346 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.886318 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-kube-api-access-7nxdl\" (UniqueName: \"kubernetes.io/projected/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-kube-api-access-7nxdl\") pod \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " Oct 31 04:20:25.886535 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.886401 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-plugins-config-volume\") pod \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\" (UID: \"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab\") " Oct 31 04:20:25.886916 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.886896 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-plugins-config-volume" pod "eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" (UID: "eb721cfe-87d4-4afd-810b-4fbaf79ea4ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:25.888976 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.888949 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-kube-api-access-7nxdl" pod "eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" (UID: "eb721cfe-87d4-4afd-810b-4fbaf79ea4ab"). InnerVolumeSpecName "kube-api-access-7nxdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:25.987066 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987009 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-openshift-service-ca\") pod \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " Oct 31 04:20:25.987066 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987049 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-metrics-certs\") pod \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " Oct 31 04:20:25.987293 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987095 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-kube-api-access-fgqtq\" (UniqueName: \"kubernetes.io/projected/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-kube-api-access-fgqtq\") pod \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " Oct 31 04:20:25.987293 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987152 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-plugins-config-volume\") pod \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " Oct 31 04:20:25.987293 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987170 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls\") pod \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\" (UID: \"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3\") " Oct 31 04:20:25.987455 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987296 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-kube-api-access-lq5s2\" (UniqueName: \"kubernetes.io/projected/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-kube-api-access-lq5s2\") pod \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\" (UID: \"c7b5b31b-e28b-4c1f-b46e-5b590b476d58\") " Oct 31 04:20:25.987455 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987415 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-openshift-service-ca" pod "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" (UID: "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:25.987606 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987559 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-plugins-config-volume" pod "c7b5b31b-e28b-4c1f-b46e-5b590b476d58" (UID: "c7b5b31b-e28b-4c1f-b46e-5b590b476d58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:25.987648 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987631 4934 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-openshift-service-ca\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:25.987683 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987647 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nxdl\" (UniqueName: \"kubernetes.io/projected/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-kube-api-access-7nxdl\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:25.987683 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.987668 4934 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab-plugins-config-volume\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:25.989617 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.989594 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-kube-api-access-lq5s2" pod "c7b5b31b-e28b-4c1f-b46e-5b590b476d58" (UID: "c7b5b31b-e28b-4c1f-b46e-5b590b476d58"). InnerVolumeSpecName "kube-api-access-lq5s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:25.989740 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.989686 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-metrics-certs" pod "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" (UID: "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:25.989905 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.989885 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-kube-api-access-fgqtq" pod "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" (UID: "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3"). InnerVolumeSpecName "kube-api-access-fgqtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:25.990056 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:25.990024 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls" pod "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" (UID: "a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:20:26.089082 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.089055 4934 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-metrics-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:26.089082 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.089080 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqtq\" (UniqueName: \"kubernetes.io/projected/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-kube-api-access-fgqtq\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:26.089326 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.089093 4934 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-plugins-config-volume\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:26.089326 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.089101 4934 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3-maas-api-tls\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:26.089326 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.089110 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq5s2\" (UniqueName: \"kubernetes.io/projected/c7b5b31b-e28b-4c1f-b46e-5b590b476d58-kube-api-access-lq5s2\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:26.147812 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.147777 4934 generic.go:356] "Generic (PLEG): container finished" podID="c7b5b31b-e28b-4c1f-b46e-5b590b476d58" containerID="f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0" exitCode=0 Oct 31 04:20:26.148002 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.147834 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" Oct 31 04:20:26.148002 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.147890 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" event={"ID":"c7b5b31b-e28b-4c1f-b46e-5b590b476d58","Type":"ContainerDied","Data":"f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0"} Oct 31 04:20:26.148002 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.147937 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp" event={"ID":"c7b5b31b-e28b-4c1f-b46e-5b590b476d58","Type":"ContainerDied","Data":"e4ce90a3556588382e8b698b619f9b892d8ed18b89bde0be23b78ed6236d08b5"} Oct 31 04:20:26.148002 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.147966 4934 scope.go:122] "RemoveContainer" containerID="f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0" Oct 31 04:20:26.150487 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.150461 4934 generic.go:356] "Generic (PLEG): container finished" podID="a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" containerID="e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1" exitCode=0 Oct 31 04:20:26.150614 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.150503 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" Oct 31 04:20:26.150614 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.150549 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" event={"ID":"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3","Type":"ContainerDied","Data":"e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1"} Oct 31 04:20:26.150614 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.150602 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl" event={"ID":"a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3","Type":"ContainerDied","Data":"f27009410c2b4ff6e6803a68fcbf4014d5a3e52d002f5165607dd203e3d39ab9"} Oct 31 04:20:26.152084 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.152062 4934 generic.go:356] "Generic (PLEG): container finished" podID="eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" containerID="5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52" exitCode=0 Oct 31 04:20:26.152198 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.152161 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" event={"ID":"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab","Type":"ContainerDied","Data":"5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52"} Oct 31 04:20:26.152295 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.152198 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" event={"ID":"eb721cfe-87d4-4afd-810b-4fbaf79ea4ab","Type":"ContainerDied","Data":"a9a1f3501e9c08dd6d92e7c59107ed9d1ac685f66d1dc734ae0579c192ceba4e"} Oct 31 04:20:26.152295 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.152277 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk" Oct 31 04:20:26.159492 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.159474 4934 scope.go:122] "RemoveContainer" containerID="f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0" Oct 31 04:20:26.159790 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:20:26.159771 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0\": container with ID starting with f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0 not found: ID does not exist" containerID="f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0" Oct 31 04:20:26.159868 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.159796 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0"} err="failed to get container status \"f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0\": rpc error: code = NotFound desc = could not find container \"f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0\": container with ID starting with f885b5475e4226eeeb8a611d172f1cf2cfe63104845ecf93fa6b0b3227e7b1f0 not found: ID does not exist" Oct 31 04:20:26.159868 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.159812 4934 scope.go:122] "RemoveContainer" containerID="e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1" Oct 31 04:20:26.168926 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.168898 4934 scope.go:122] "RemoveContainer" containerID="e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1" Oct 31 04:20:26.169176 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:20:26.169156 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1\": container with ID starting with e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1 not found: ID does not exist" containerID="e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1" Oct 31 04:20:26.169277 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.169183 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1"} err="failed to get container status \"e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1\": rpc error: code = NotFound desc = could not find container \"e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1\": container with ID starting with e95d031e37440e4ce9dd5d66d54acf40433ec86fb5a2a16f45b67e1eabc028a1 not found: ID does not exist" Oct 31 04:20:26.169277 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.169200 4934 scope.go:122] "RemoveContainer" containerID="5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52" Oct 31 04:20:26.176791 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.176775 4934 scope.go:122] "RemoveContainer" containerID="5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52" Oct 31 04:20:26.177027 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:20:26.177009 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52\": container with ID starting with 5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52 not found: ID does not exist" containerID="5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52" Oct 31 04:20:26.177091 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.177036 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52"} err="failed to get container status \"5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52\": rpc error: code = NotFound desc = could not find container \"5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52\": container with ID starting with 5981dba9d12e53a3b72aa2328bd75208723678f4f3eb980ae90fd5bcec084d52 not found: ID does not exist" Oct 31 04:20:26.311901 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.311866 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk"] Oct 31 04:20:26.316812 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.316788 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a058534b-f69db68c-dzhnk"] Oct 31 04:20:26.426752 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.426717 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl"] Oct 31 04:20:26.431126 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.431091 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a058534b-ff6f9c5b7-n94pl"] Oct 31 04:20:26.541204 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.541167 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp"] Oct 31 04:20:26.545828 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.545795 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a058534b-5d75677b5f-x7tfp"] Oct 31 04:20:26.822256 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.822210 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" path="/var/lib/kubelet/pods/a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3/volumes" Oct 31 04:20:26.822609 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.822595 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b5b31b-e28b-4c1f-b46e-5b590b476d58" path="/var/lib/kubelet/pods/c7b5b31b-e28b-4c1f-b46e-5b590b476d58/volumes" Oct 31 04:20:26.822917 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:26.822905 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" path="/var/lib/kubelet/pods/eb721cfe-87d4-4afd-810b-4fbaf79ea4ab/volumes" Oct 31 04:20:31.860486 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.860440 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv"] Oct 31 04:20:31.861692 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.861665 4934 cpu_manager.go:414] "containerMap: removing container" podUID="c7b5b31b-e28b-4c1f-b46e-5b590b476d58" containerName="payload-pre-processing" Oct 31 04:20:31.861692 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.861690 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b5b31b-e28b-4c1f-b46e-5b590b476d58" containerName="payload-pre-processing" Oct 31 04:20:31.861866 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.861725 4934 cpu_manager.go:414] "containerMap: removing container" podUID="eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" containerName="payload-processing" Oct 31 04:20:31.861866 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.861734 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" containerName="payload-processing" Oct 31 04:20:31.861866 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.861803 4934 cpu_manager.go:414] "containerMap: removing container" podUID="a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" containerName="maas-api" Oct 31 04:20:31.861866 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.861810 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" containerName="maas-api" Oct 31 04:20:31.862602 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.862580 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bd6a9a-c69a-4cd6-b43b-cba121ee3ac3" containerName="maas-api" Oct 31 04:20:31.862709 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.862615 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b5b31b-e28b-4c1f-b46e-5b590b476d58" containerName="payload-pre-processing" Oct 31 04:20:31.862709 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.862628 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb721cfe-87d4-4afd-810b-4fbaf79ea4ab" containerName="payload-processing" Oct 31 04:20:31.868031 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.868007 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:31.921212 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.921183 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-0bda5d-openshift-default-dockercfg-bqkpm\"" Oct 31 04:20:31.972428 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:31.972390 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv"] Oct 31 04:20:32.040574 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040540 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-socket\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040751 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040586 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istiod-ca-cert\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040751 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040677 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-envoy\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040751 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040706 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-podinfo\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040751 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040722 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7z4v\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-kube-api-access-b7z4v\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040751 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040747 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-data\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040916 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040887 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-certs\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040960 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040945 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-token\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.040993 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.040972 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-credential-socket\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.141875 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.141792 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-token\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.141875 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.141829 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-credential-socket\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.141875 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.141859 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-socket\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142176 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.141886 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istiod-ca-cert\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142176 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.141926 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-envoy\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142176 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.141948 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-podinfo\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142176 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.141967 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7z4v\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-kube-api-access-b7z4v\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142176 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.142000 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-data\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142176 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.142037 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-certs\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142499 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.142319 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-credential-socket\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142499 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.142362 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-socket\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142499 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.142441 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-data\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.142668 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.142647 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-certs\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.143017 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.142993 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istiod-ca-cert\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.144647 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.144626 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-envoy\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.144772 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.144752 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-podinfo\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.203447 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.203404 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-token\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.203600 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.203519 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7z4v\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-kube-api-access-b7z4v\") pod \"e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.481762 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.481670 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:32.908827 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.908628 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv"] Oct 31 04:20:32.912296 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:20:32.912266 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2d70b5_1712_4feb_9bb3_09915dbfd906.slice/crio-f347578a889278f9f463e37cc952120e742d6223485dd67ae54d7186a50c687e WatchSource:0}: Error finding container f347578a889278f9f463e37cc952120e742d6223485dd67ae54d7186a50c687e: Status 404 returned error can't find the container with id f347578a889278f9f463e37cc952120e742d6223485dd67ae54d7186a50c687e Oct 31 04:20:32.914483 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.914451 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:20:32.914569 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.914519 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:20:32.914569 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:32.914548 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:20:33.184989 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:33.184892 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" event={"ID":"0b2d70b5-1712-4feb-9bb3-09915dbfd906","Type":"ContainerStarted","Data":"222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb"} Oct 31 04:20:33.184989 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:33.184952 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" event={"ID":"0b2d70b5-1712-4feb-9bb3-09915dbfd906","Type":"ContainerStarted","Data":"f347578a889278f9f463e37cc952120e742d6223485dd67ae54d7186a50c687e"} Oct 31 04:20:33.306089 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:33.306016 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" podStartSLOduration=2.306001078 podStartE2EDuration="2.306001078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:33.304779528 +0000 UTC m=+3666.623522205" watchObservedRunningTime="2026-10-31 04:20:33.306001078 +0000 UTC m=+3666.624743704" Oct 31 04:20:33.481935 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:33.481840 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:33.487315 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:33.487275 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:34.189649 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:34.189616 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:34.190656 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:34.190636 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:20:39.883983 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:39.883940 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx"] Oct 31 04:20:39.887650 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:39.887629 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:39.914599 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:39.914555 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzcd\" (UniqueName: \"kubernetes.io/projected/45fd3a93-9a71-497b-9bd8-97715d892719-kube-api-access-pmzcd\") pod \"payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:39.914809 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:39.914725 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fd3a93-9a71-497b-9bd8-97715d892719-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:39.990339 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:39.990293 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-0bda5d\"" Oct 31 04:20:39.990718 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:39.990663 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-0bda5d-dockercfg-zcvzw\"" Oct 31 04:20:39.992489 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:39.992463 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx"] Oct 31 04:20:40.016171 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.016133 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fd3a93-9a71-497b-9bd8-97715d892719-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:40.016171 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.016177 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzcd\" (UniqueName: \"kubernetes.io/projected/45fd3a93-9a71-497b-9bd8-97715d892719-kube-api-access-pmzcd\") pod \"payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:40.016942 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.016913 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fd3a93-9a71-497b-9bd8-97715d892719-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:40.043052 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.043012 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg"] Oct 31 04:20:40.046581 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.046561 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.077909 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.077866 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzcd\" (UniqueName: \"kubernetes.io/projected/45fd3a93-9a71-497b-9bd8-97715d892719-kube-api-access-pmzcd\") pod \"payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:40.100219 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.100181 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg"] Oct 31 04:20:40.117766 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.117728 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a371d866-32b9-455a-91ff-126428bdf781-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.117948 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.117865 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpksl\" (UniqueName: \"kubernetes.io/projected/a371d866-32b9-455a-91ff-126428bdf781-kube-api-access-kpksl\") pod \"payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.200716 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.200614 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:40.218836 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.218788 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpksl\" (UniqueName: \"kubernetes.io/projected/a371d866-32b9-455a-91ff-126428bdf781-kube-api-access-kpksl\") pod \"payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.219022 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.218898 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a371d866-32b9-455a-91ff-126428bdf781-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.219801 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.219773 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a371d866-32b9-455a-91ff-126428bdf781-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.282336 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.282265 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpksl\" (UniqueName: \"kubernetes.io/projected/a371d866-32b9-455a-91ff-126428bdf781-kube-api-access-kpksl\") pod \"payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.360555 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.360496 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:40.589023 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.588891 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx"] Oct 31 04:20:40.591862 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:20:40.591831 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fd3a93_9a71_497b_9bd8_97715d892719.slice/crio-bfcc535dd1f82a68fd394d7ef798ba5fe28dbd1ce88a6b2b75b91af657509684 WatchSource:0}: Error finding container bfcc535dd1f82a68fd394d7ef798ba5fe28dbd1ce88a6b2b75b91af657509684: Status 404 returned error can't find the container with id bfcc535dd1f82a68fd394d7ef798ba5fe28dbd1ce88a6b2b75b91af657509684 Oct 31 04:20:40.738686 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:40.738485 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg"] Oct 31 04:20:40.741293 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:20:40.741264 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda371d866_32b9_455a_91ff_126428bdf781.slice/crio-1e193b266f8544b91960579e7a7f175c471978ea1d76582e41635fd5913a80db WatchSource:0}: Error finding container 1e193b266f8544b91960579e7a7f175c471978ea1d76582e41635fd5913a80db: Status 404 returned error can't find the container with id 1e193b266f8544b91960579e7a7f175c471978ea1d76582e41635fd5913a80db Oct 31 04:20:41.233308 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.233260 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" event={"ID":"a371d866-32b9-455a-91ff-126428bdf781","Type":"ContainerStarted","Data":"7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad"} Oct 31 04:20:41.233763 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.233367 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" event={"ID":"a371d866-32b9-455a-91ff-126428bdf781","Type":"ContainerStarted","Data":"1e193b266f8544b91960579e7a7f175c471978ea1d76582e41635fd5913a80db"} Oct 31 04:20:41.233763 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.233396 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:41.235156 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.235122 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" event={"ID":"45fd3a93-9a71-497b-9bd8-97715d892719","Type":"ContainerStarted","Data":"a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12"} Oct 31 04:20:41.235304 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.235177 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" event={"ID":"45fd3a93-9a71-497b-9bd8-97715d892719","Type":"ContainerStarted","Data":"bfcc535dd1f82a68fd394d7ef798ba5fe28dbd1ce88a6b2b75b91af657509684"} Oct 31 04:20:41.235304 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.235198 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:20:41.350272 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.350178 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" podStartSLOduration=2.350163619 podStartE2EDuration="2.350163619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:41.348928009 +0000 UTC m=+3674.667670635" watchObservedRunningTime="2026-10-31 04:20:41.350163619 +0000 UTC m=+3674.668906249" Oct 31 04:20:41.466799 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:41.466728 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" podStartSLOduration=2.466704368 podStartE2EDuration="2.466704368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:41.464406017 +0000 UTC m=+3674.783148645" watchObservedRunningTime="2026-10-31 04:20:41.466704368 +0000 UTC m=+3674.785446996" Oct 31 04:20:49.165277 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:20:49.165214 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:20:52.243335 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:52.243291 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:20:52.243725 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:20:52.243374 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:21:00.340938 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:00.340901 4934 generic.go:356] "Generic (PLEG): container finished" podID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerID="222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb" exitCode=0 Oct 31 04:21:00.341415 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:00.340965 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" event={"ID":"0b2d70b5-1712-4feb-9bb3-09915dbfd906","Type":"ContainerDied","Data":"222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb"} Oct 31 04:21:00.341460 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:00.341419 4934 scope.go:122] "RemoveContainer" containerID="222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb" Oct 31 04:21:00.342680 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:00.342642 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:21:00.342754 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:00.342735 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:21:00.342819 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:00.342805 4934 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891988Ki","pods":"250"} Oct 31 04:21:01.347470 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:01.347430 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" event={"ID":"0b2d70b5-1712-4feb-9bb3-09915dbfd906","Type":"ContainerStarted","Data":"dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073"} Oct 31 04:21:01.482649 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:01.482605 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:21:01.483793 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:01.483764 4934 patch_prober.go:29] interesting pod/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 04:21:01.483945 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:01.483820 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 04:21:02.482749 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:02.482689 4934 patch_prober.go:29] interesting pod/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 04:21:02.483350 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:02.482769 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 04:21:03.482532 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:03.482492 4934 patch_prober.go:29] interesting pod/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 04:21:03.482738 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:03.482549 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 04:21:04.189838 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:04.189801 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:21:04.483053 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:04.482955 4934 patch_prober.go:29] interesting pod/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 04:21:04.483053 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:04.483018 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 04:21:05.482460 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:05.482425 4934 patch_prober.go:29] interesting pod/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 04:21:05.482890 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:05.482482 4934 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 04:21:06.525131 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:06.525098 4934 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:21:06.526305 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:21:06.526279 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:21:19.161393 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:21:19.161359 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:21:49.161912 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:21:49.161832 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:22:19.163683 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:22:19.163643 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:22:33.256716 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:33.256682 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg"] Oct 31 04:22:33.360223 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:33.360195 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx"] Oct 31 04:22:38.257124 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.257073 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" podUID="a371d866-32b9-455a-91ff-126428bdf781" containerName="payload-processing" containerID="cri-o://7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad" gracePeriod=25 Oct 31 04:22:38.361043 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.361003 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" podUID="45fd3a93-9a71-497b-9bd8-97715d892719" containerName="payload-pre-processing" containerID="cri-o://a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12" gracePeriod=25 Oct 31 04:22:38.509587 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.509514 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:22:38.599500 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.599464 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a371d866-32b9-455a-91ff-126428bdf781-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a371d866-32b9-455a-91ff-126428bdf781-plugins-config-volume\") pod \"a371d866-32b9-455a-91ff-126428bdf781\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " Oct 31 04:22:38.599700 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.599565 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a371d866-32b9-455a-91ff-126428bdf781-kube-api-access-kpksl\" (UniqueName: \"kubernetes.io/projected/a371d866-32b9-455a-91ff-126428bdf781-kube-api-access-kpksl\") pod \"a371d866-32b9-455a-91ff-126428bdf781\" (UID: \"a371d866-32b9-455a-91ff-126428bdf781\") " Oct 31 04:22:38.599905 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.599877 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a371d866-32b9-455a-91ff-126428bdf781-plugins-config-volume" pod "a371d866-32b9-455a-91ff-126428bdf781" (UID: "a371d866-32b9-455a-91ff-126428bdf781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:38.601781 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.601758 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a371d866-32b9-455a-91ff-126428bdf781-kube-api-access-kpksl" pod "a371d866-32b9-455a-91ff-126428bdf781" (UID: "a371d866-32b9-455a-91ff-126428bdf781"). InnerVolumeSpecName "kube-api-access-kpksl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:38.611566 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.611549 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:22:38.700838 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.700812 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fd3a93-9a71-497b-9bd8-97715d892719-kube-api-access-pmzcd\" (UniqueName: \"kubernetes.io/projected/45fd3a93-9a71-497b-9bd8-97715d892719-kube-api-access-pmzcd\") pod \"45fd3a93-9a71-497b-9bd8-97715d892719\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " Oct 31 04:22:38.700982 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.700841 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45fd3a93-9a71-497b-9bd8-97715d892719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fd3a93-9a71-497b-9bd8-97715d892719-plugins-config-volume\") pod \"45fd3a93-9a71-497b-9bd8-97715d892719\" (UID: \"45fd3a93-9a71-497b-9bd8-97715d892719\") " Oct 31 04:22:38.701086 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.701065 4934 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a371d866-32b9-455a-91ff-126428bdf781-plugins-config-volume\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:38.701086 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.701083 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpksl\" (UniqueName: \"kubernetes.io/projected/a371d866-32b9-455a-91ff-126428bdf781-kube-api-access-kpksl\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:38.701223 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.701173 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fd3a93-9a71-497b-9bd8-97715d892719-plugins-config-volume" pod "45fd3a93-9a71-497b-9bd8-97715d892719" (UID: "45fd3a93-9a71-497b-9bd8-97715d892719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:38.702858 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.702843 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fd3a93-9a71-497b-9bd8-97715d892719-kube-api-access-pmzcd" pod "45fd3a93-9a71-497b-9bd8-97715d892719" (UID: "45fd3a93-9a71-497b-9bd8-97715d892719"). InnerVolumeSpecName "kube-api-access-pmzcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:38.799165 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.799099 4934 generic.go:356] "Generic (PLEG): container finished" podID="a371d866-32b9-455a-91ff-126428bdf781" containerID="7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad" exitCode=0 Oct 31 04:22:38.799165 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.799150 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" Oct 31 04:22:38.799367 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.799182 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" event={"ID":"a371d866-32b9-455a-91ff-126428bdf781","Type":"ContainerDied","Data":"7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad"} Oct 31 04:22:38.799367 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.799294 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg" event={"ID":"a371d866-32b9-455a-91ff-126428bdf781","Type":"ContainerDied","Data":"1e193b266f8544b91960579e7a7f175c471978ea1d76582e41635fd5913a80db"} Oct 31 04:22:38.799367 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.799322 4934 scope.go:122] "RemoveContainer" containerID="7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad" Oct 31 04:22:38.800523 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.800503 4934 generic.go:356] "Generic (PLEG): container finished" podID="45fd3a93-9a71-497b-9bd8-97715d892719" containerID="a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12" exitCode=0 Oct 31 04:22:38.800609 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.800553 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" Oct 31 04:22:38.800609 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.800585 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" event={"ID":"45fd3a93-9a71-497b-9bd8-97715d892719","Type":"ContainerDied","Data":"a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12"} Oct 31 04:22:38.800719 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.800645 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx" event={"ID":"45fd3a93-9a71-497b-9bd8-97715d892719","Type":"ContainerDied","Data":"bfcc535dd1f82a68fd394d7ef798ba5fe28dbd1ce88a6b2b75b91af657509684"} Oct 31 04:22:38.801668 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.801647 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzcd\" (UniqueName: \"kubernetes.io/projected/45fd3a93-9a71-497b-9bd8-97715d892719-kube-api-access-pmzcd\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:38.801743 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.801674 4934 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fd3a93-9a71-497b-9bd8-97715d892719-plugins-config-volume\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:38.807798 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.807783 4934 scope.go:122] "RemoveContainer" containerID="7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad" Oct 31 04:22:38.808006 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:22:38.807989 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad\": container with ID starting with 7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad not found: ID does not exist" containerID="7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad" Oct 31 04:22:38.808078 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.808011 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad"} err="failed to get container status \"7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad\": rpc error: code = NotFound desc = could not find container \"7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad\": container with ID starting with 7a065dbf4b270dec53a5d10d61127aa518883c5481049d17081a7c78542c16ad not found: ID does not exist" Oct 31 04:22:38.808078 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.808027 4934 scope.go:122] "RemoveContainer" containerID="a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12" Oct 31 04:22:38.816710 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.816696 4934 scope.go:122] "RemoveContainer" containerID="a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12" Oct 31 04:22:38.816962 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:22:38.816943 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12\": container with ID starting with a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12 not found: ID does not exist" containerID="a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12" Oct 31 04:22:38.817027 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.816968 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12"} err="failed to get container status \"a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12\": rpc error: code = NotFound desc = could not find container \"a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12\": container with ID starting with a10e79be27eea5e68d1589ee9c210218c66b3c7a9c81917fc869fa9a60f77f12 not found: ID does not exist" Oct 31 04:22:38.964440 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.964417 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg"] Oct 31 04:22:38.968707 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:38.968684 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0bda5d-58988557b7-8llzg"] Oct 31 04:22:39.079187 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:39.079165 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx"] Oct 31 04:22:39.082614 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:39.082596 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0bda5d-66956f8f7f-8cvkx"] Oct 31 04:22:40.819884 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:40.819855 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fd3a93-9a71-497b-9bd8-97715d892719" path="/var/lib/kubelet/pods/45fd3a93-9a71-497b-9bd8-97715d892719/volumes" Oct 31 04:22:40.820260 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:40.820189 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a371d866-32b9-455a-91ff-126428bdf781" path="/var/lib/kubelet/pods/a371d866-32b9-455a-91ff-126428bdf781/volumes" Oct 31 04:22:49.161967 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:22:49.161932 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:22:53.573113 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:53.573080 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv"] Oct 31 04:22:53.573671 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:53.573341 4934 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" containerID="cri-o://dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073" gracePeriod=30 Oct 31 04:22:58.820677 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.820653 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:22:58.869569 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869515 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-podinfo\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869569 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869545 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-kube-api-access-b7z4v\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-kube-api-access-b7z4v\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869569 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869565 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-data\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869798 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869588 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-certs\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869798 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869634 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-credential-socket\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869798 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869658 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-envoy\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869798 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869695 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-token\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-token\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869798 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869725 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-socket\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.869798 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869766 4934 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istiod-ca-cert\") pod \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\" (UID: \"0b2d70b5-1712-4feb-9bb3-09915dbfd906\") " Oct 31 04:22:58.870123 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869891 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-data" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:58.870123 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869949 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-certs" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:58.870123 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.869970 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-credential-socket" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:58.870123 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.870080 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-socket" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:58.870382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.870211 4934 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.870382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.870261 4934 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-data\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.870382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.870276 4934 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-workload-certs\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.870382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.870289 4934 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-credential-socket\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.870382 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.870292 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istiod-ca-cert" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:58.871822 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.871801 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-kube-api-access-b7z4v" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "kube-api-access-b7z4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:58.872038 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.872024 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-token" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:58.872289 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.872271 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-envoy" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:22:58.872359 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.872317 4934 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-podinfo" pod "0b2d70b5-1712-4feb-9bb3-09915dbfd906" (UID: "0b2d70b5-1712-4feb-9bb3-09915dbfd906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:22:58.887831 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.887804 4934 generic.go:356] "Generic (PLEG): container finished" podID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerID="dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073" exitCode=0 Oct 31 04:22:58.887946 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.887847 4934 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" Oct 31 04:22:58.887946 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.887903 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" event={"ID":"0b2d70b5-1712-4feb-9bb3-09915dbfd906","Type":"ContainerDied","Data":"dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073"} Oct 31 04:22:58.888025 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.887963 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv" event={"ID":"0b2d70b5-1712-4feb-9bb3-09915dbfd906","Type":"ContainerDied","Data":"f347578a889278f9f463e37cc952120e742d6223485dd67ae54d7186a50c687e"} Oct 31 04:22:58.888025 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.887986 4934 scope.go:122] "RemoveContainer" containerID="dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073" Oct 31 04:22:58.896312 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.896299 4934 scope.go:122] "RemoveContainer" containerID="222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb" Oct 31 04:22:58.903892 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.903874 4934 scope.go:122] "RemoveContainer" containerID="dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073" Oct 31 04:22:58.904151 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:22:58.904128 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073\": container with ID starting with dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073 not found: ID does not exist" containerID="dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073" Oct 31 04:22:58.904260 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.904157 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073"} err="failed to get container status \"dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073\": rpc error: code = NotFound desc = could not find container \"dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073\": container with ID starting with dc1a199cb43a4a5a5f9cb15b256a7fd57ab78ed95e5633d1a43df63f675fa073 not found: ID does not exist" Oct 31 04:22:58.904260 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.904174 4934 scope.go:122] "RemoveContainer" containerID="222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb" Oct 31 04:22:58.904439 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:22:58.904421 4934 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb\": container with ID starting with 222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb not found: ID does not exist" containerID="222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb" Oct 31 04:22:58.904496 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.904450 4934 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb"} err="failed to get container status \"222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb\": rpc error: code = NotFound desc = could not find container \"222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb\": container with ID starting with 222c0f0f86be2f5e0a1f73702f5e5d939782f0661d78088cde15d9980545cbeb not found: ID does not exist" Oct 31 04:22:58.971415 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.971393 4934 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-envoy\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.971415 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.971415 4934 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-token\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.971553 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.971429 4934 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istiod-ca-cert\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.971553 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.971443 4934 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2d70b5-1712-4feb-9bb3-09915dbfd906-istio-podinfo\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:58.971553 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:58.971456 4934 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7z4v\" (UniqueName: \"kubernetes.io/projected/0b2d70b5-1712-4feb-9bb3-09915dbfd906-kube-api-access-b7z4v\") on node \"ip-10-0-138-238.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:59.103304 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:59.103256 4934 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv"] Oct 31 04:22:59.109909 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:22:59.109884 4934 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0bda5d-openshift-default-844d9b5cc7-qtnpv"] Oct 31 04:23:00.820725 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:23:00.820694 4934 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" path="/var/lib/kubelet/pods/0b2d70b5-1712-4feb-9bb3-09915dbfd906/volumes" Oct 31 04:23:19.163179 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:23:19.163138 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:23:49.163165 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:23:49.163131 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:24:18.006219 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:18.006190 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qbv2b_3a35468d-dc35-4c8b-9afe-e59d5c2b39be/manager/0.log" Oct 31 04:24:18.337544 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:18.337519 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768d4c8b4f-2k2s9_bb9a6c42-db24-41bc-a5eb-666e1c180761/manager/0.log" Oct 31 04:24:19.162637 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:24:19.162596 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:24:20.285851 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:20.282800 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796fbdf5b4-xndbd_260917c8-b794-455a-b94e-37f2956f8b03/maas-api/0.log" Oct 31 04:24:23.987139 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:23.987109 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nhqmq_7e751df4-44e7-4fd6-856b-34586f7a28fa/manager/0.log" Oct 31 04:24:26.108683 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:26.108650 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-764498d796-fmmn5_8f03c061-d61a-4f36-9a03-4bdd462df099/kube-auth-proxy/0.log" Oct 31 04:24:27.095241 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:27.095189 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-kll7s_a2e58daa-d4a4-45a5-b9ae-d7906f70c916/istio-proxy/0.log" Oct 31 04:24:27.807803 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:27.807778 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:24:27.867586 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:27.867563 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b54xh_9bdf0731-bef7-4c9c-a5c3-09eff5aef394/console-operator/3.log" Oct 31 04:24:27.883834 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:27.883808 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-q9mpw_b9444d14-3d85-439f-a7ef-751d9fee021e/istio-proxy/0.log" Oct 31 04:24:28.117306 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:28.117244 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-s5kcb_5453941c-2d09-4ff9-aaaf-cc0e162876e0/istio-proxy/0.log" Oct 31 04:24:29.526872 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:29.526842 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d69c76ccf-dccpx_7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/router/0.log" Oct 31 04:24:43.213386 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:43.213317 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qbv2b_3a35468d-dc35-4c8b-9afe-e59d5c2b39be/manager/0.log" Oct 31 04:24:43.486987 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:43.486911 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768d4c8b4f-2k2s9_bb9a6c42-db24-41bc-a5eb-666e1c180761/manager/0.log" Oct 31 04:24:45.164871 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:45.164844 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796fbdf5b4-xndbd_260917c8-b794-455a-b94e-37f2956f8b03/maas-api/0.log" Oct 31 04:24:48.592319 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:48.592283 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nhqmq_7e751df4-44e7-4fd6-856b-34586f7a28fa/manager/0.log" Oct 31 04:24:49.162543 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:24:49.162511 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:24:50.498932 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:50.498881 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-764498d796-fmmn5_8f03c061-d61a-4f36-9a03-4bdd462df099/kube-auth-proxy/0.log" Oct 31 04:24:51.507863 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:51.507839 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-kll7s_a2e58daa-d4a4-45a5-b9ae-d7906f70c916/istio-proxy/0.log" Oct 31 04:24:52.290293 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:52.290260 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-q9mpw_b9444d14-3d85-439f-a7ef-751d9fee021e/istio-proxy/0.log" Oct 31 04:24:52.543360 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:52.543297 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-s5kcb_5453941c-2d09-4ff9-aaaf-cc0e162876e0/istio-proxy/0.log" Oct 31 04:24:53.983984 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:24:53.983953 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d69c76ccf-dccpx_7c86d4eb-4dbf-4daf-95a3-62d73fc5210b/router/0.log" Oct 31 04:25:19.162363 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:25:19.162321 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:25:27.624716 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:25:27.624687 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768d4c8b4f-2k2s9_bb9a6c42-db24-41bc-a5eb-666e1c180761/manager/0.log" Oct 31 04:25:28.204088 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:25:28.204059 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768d4c8b4f-2k2s9_bb9a6c42-db24-41bc-a5eb-666e1c180761/manager/0.log" Oct 31 04:25:49.162147 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:25:49.162098 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:26:06.772928 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:06.772831 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768d4c8b4f-2k2s9_bb9a6c42-db24-41bc-a5eb-666e1c180761/manager/0.log" Oct 31 04:26:07.344343 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:07.344310 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768d4c8b4f-2k2s9_bb9a6c42-db24-41bc-a5eb-666e1c180761/manager/0.log" Oct 31 04:26:08.672029 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.671995 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xr5kx/must-gather-xgl8g"] Oct 31 04:26:08.672420 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672339 4934 cpu_manager.go:414] "containerMap: removing container" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" Oct 31 04:26:08.672420 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672350 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" Oct 31 04:26:08.672420 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672370 4934 cpu_manager.go:414] "containerMap: removing container" podUID="45fd3a93-9a71-497b-9bd8-97715d892719" containerName="payload-pre-processing" Oct 31 04:26:08.672420 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672375 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fd3a93-9a71-497b-9bd8-97715d892719" containerName="payload-pre-processing" Oct 31 04:26:08.672420 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672382 4934 cpu_manager.go:414] "containerMap: removing container" podUID="a371d866-32b9-455a-91ff-126428bdf781" containerName="payload-processing" Oct 31 04:26:08.672420 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672388 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a371d866-32b9-455a-91ff-126428bdf781" containerName="payload-processing" Oct 31 04:26:08.672608 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672442 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" Oct 31 04:26:08.672608 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672452 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fd3a93-9a71-497b-9bd8-97715d892719" containerName="payload-pre-processing" Oct 31 04:26:08.672608 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672458 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="a371d866-32b9-455a-91ff-126428bdf781" containerName="payload-processing" Oct 31 04:26:08.672608 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672528 4934 cpu_manager.go:414] "containerMap: removing container" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" Oct 31 04:26:08.672608 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672533 4934 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" Oct 31 04:26:08.672608 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.672578 4934 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2d70b5-1712-4feb-9bb3-09915dbfd906" containerName="istio-proxy" Oct 31 04:26:08.675415 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.675400 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:08.728285 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.728259 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr5kx\"/\"openshift-service-ca.crt\"" Oct 31 04:26:08.777660 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.777635 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j48r\" (UniqueName: \"kubernetes.io/projected/60e1d725-e4ab-4c2b-8625-832fb4335efd-kube-api-access-7j48r\") pod \"must-gather-xgl8g\" (UID: \"60e1d725-e4ab-4c2b-8625-832fb4335efd\") " pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:08.777765 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.777682 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60e1d725-e4ab-4c2b-8625-832fb4335efd-must-gather-output\") pod \"must-gather-xgl8g\" (UID: \"60e1d725-e4ab-4c2b-8625-832fb4335efd\") " pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:08.777995 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.777980 4934 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr5kx\"/\"kube-root-ca.crt\"" Oct 31 04:26:08.778037 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.778016 4934 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xr5kx\"/\"default-dockercfg-mqfbg\"" Oct 31 04:26:08.779377 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.779360 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/must-gather-xgl8g"] Oct 31 04:26:08.878148 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.878127 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60e1d725-e4ab-4c2b-8625-832fb4335efd-must-gather-output\") pod \"must-gather-xgl8g\" (UID: \"60e1d725-e4ab-4c2b-8625-832fb4335efd\") " pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:08.878321 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.878207 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j48r\" (UniqueName: \"kubernetes.io/projected/60e1d725-e4ab-4c2b-8625-832fb4335efd-kube-api-access-7j48r\") pod \"must-gather-xgl8g\" (UID: \"60e1d725-e4ab-4c2b-8625-832fb4335efd\") " pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:08.878478 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.878459 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60e1d725-e4ab-4c2b-8625-832fb4335efd-must-gather-output\") pod \"must-gather-xgl8g\" (UID: \"60e1d725-e4ab-4c2b-8625-832fb4335efd\") " pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:08.939030 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.938972 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j48r\" (UniqueName: \"kubernetes.io/projected/60e1d725-e4ab-4c2b-8625-832fb4335efd-kube-api-access-7j48r\") pod \"must-gather-xgl8g\" (UID: \"60e1d725-e4ab-4c2b-8625-832fb4335efd\") " pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:08.999807 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:08.999782 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/must-gather-xgl8g" Oct 31 04:26:09.378416 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:09.378388 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/must-gather-xgl8g"] Oct 31 04:26:09.381055 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:26:09.381024 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e1d725_e4ab_4c2b_8625_832fb4335efd.slice/crio-0ef3911e9e89c807408aba75fad8b21c9c665b371c4d666c141d4ef3ff5abf72 WatchSource:0}: Error finding container 0ef3911e9e89c807408aba75fad8b21c9c665b371c4d666c141d4ef3ff5abf72: Status 404 returned error can't find the container with id 0ef3911e9e89c807408aba75fad8b21c9c665b371c4d666c141d4ef3ff5abf72 Oct 31 04:26:09.382932 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:09.382911 4934 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:26:09.749458 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:09.749369 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/must-gather-xgl8g" event={"ID":"60e1d725-e4ab-4c2b-8625-832fb4335efd","Type":"ContainerStarted","Data":"0ef3911e9e89c807408aba75fad8b21c9c665b371c4d666c141d4ef3ff5abf72"} Oct 31 04:26:11.759934 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:11.759899 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/must-gather-xgl8g" event={"ID":"60e1d725-e4ab-4c2b-8625-832fb4335efd","Type":"ContainerStarted","Data":"e82283a654a88b02dab4226bdbacc0ef10ed18189c781253e260ebbc22ae9af9"} Oct 31 04:26:11.760366 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:11.759941 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/must-gather-xgl8g" event={"ID":"60e1d725-e4ab-4c2b-8625-832fb4335efd","Type":"ContainerStarted","Data":"17033a4ef7be901348f4649a09a6b760059e865d2f843fad47b24f6d8cd7a07e"} Oct 31 04:26:11.877900 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:11.877844 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xr5kx/must-gather-xgl8g" podStartSLOduration=2.604833981 podStartE2EDuration="3.877829342s" totalImagesPullingTime="1.272995361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:11.876537798 +0000 UTC m=+4005.195280424" watchObservedRunningTime="2026-10-31 04:26:11.877829342 +0000 UTC m=+4005.196571969" Oct 31 04:26:17.637250 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:17.637093 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5ngpx_3eefb160-a377-431f-be0a-c2bea6ecb352/global-pull-secret-syncer/0.log" Oct 31 04:26:19.202082 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:26:19.198207 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:26:19.484023 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:19.483837 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mdl5l_15c8c5bb-10f1-4894-b45d-34b2209f9676/connection-checker/0.log" Oct 31 04:26:19.918239 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:19.918197 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mdl5v_69dbc1e2-7632-40e1-8e81-79cf08b42d67/connection-checker/0.log" Oct 31 04:26:20.350903 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:20.350818 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pkgt6_4cc36af9-eaa8-40a3-b642-cc7aaf970ed6/connection-checker/0.log" Oct 31 04:26:21.773472 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:21.773441 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vgq6v_ad09c972-be06-48b1-84b0-a25fc1c645ba/konnectivity-agent/0.log" Oct 31 04:26:22.622971 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:22.622944 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-238.us-west-1.compute.internal_0de19d3008b27f6bbf6579b9645a2ecb/haproxy/0.log" Oct 31 04:26:27.554510 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.554477 4934 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph"] Oct 31 04:26:27.561931 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.561903 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.615762 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.615732 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph"] Oct 31 04:26:27.652937 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.652907 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-podres\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.652937 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.652937 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-lib-modules\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.653137 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.652960 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdq8d\" (UniqueName: \"kubernetes.io/projected/fe3344cd-c80a-4b2a-9862-991534baae0b-kube-api-access-fdq8d\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.653137 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.653087 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-proc\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.653211 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.653163 4934 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-sys\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754594 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754558 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-proc\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754764 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754624 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-sys\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754764 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754651 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-podres\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754764 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754665 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-lib-modules\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754764 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754689 4934 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdq8d\" (UniqueName: \"kubernetes.io/projected/fe3344cd-c80a-4b2a-9862-991534baae0b-kube-api-access-fdq8d\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754764 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754707 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-proc\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754764 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754707 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-sys\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754977 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754776 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-lib-modules\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.754977 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.754778 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe3344cd-c80a-4b2a-9862-991534baae0b-podres\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.815984 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.815947 4934 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdq8d\" (UniqueName: \"kubernetes.io/projected/fe3344cd-c80a-4b2a-9862-991534baae0b-kube-api-access-fdq8d\") pod \"perf-node-gather-daemonset-nshph\" (UID: \"fe3344cd-c80a-4b2a-9862-991534baae0b\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:27.875248 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:27.875195 4934 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:28.257353 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:28.257302 4934 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph"] Oct 31 04:26:28.260017 ip-10-0-138-238 kubenswrapper[4934]: W1031 04:26:28.259980 4934 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe3344cd_c80a_4b2a_9862_991534baae0b.slice/crio-827df012cef466af3a18986748ede44c0aec2ac93ff9cd5090c58bc1a8c9f521 WatchSource:0}: Error finding container 827df012cef466af3a18986748ede44c0aec2ac93ff9cd5090c58bc1a8c9f521: Status 404 returned error can't find the container with id 827df012cef466af3a18986748ede44c0aec2ac93ff9cd5090c58bc1a8c9f521 Oct 31 04:26:28.892161 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:28.892115 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" event={"ID":"fe3344cd-c80a-4b2a-9862-991534baae0b","Type":"ContainerStarted","Data":"71a3c4ccbb68ddf08294565803378f05354d6bce35fa20e1dafe923dff94c000"} Oct 31 04:26:28.892670 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:28.892187 4934 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" event={"ID":"fe3344cd-c80a-4b2a-9862-991534baae0b","Type":"ContainerStarted","Data":"827df012cef466af3a18986748ede44c0aec2ac93ff9cd5090c58bc1a8c9f521"} Oct 31 04:26:28.892670 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:28.892305 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:28.916087 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:28.916061 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nhqmq_7e751df4-44e7-4fd6-856b-34586f7a28fa/manager/0.log" Oct 31 04:26:29.008576 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:29.008525 4934 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" podStartSLOduration=2.008512171 podStartE2EDuration="2.008512171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:29.007529505 +0000 UTC m=+4022.326272133" watchObservedRunningTime="2026-10-31 04:26:29.008512171 +0000 UTC m=+4022.327254796" Oct 31 04:26:34.907083 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:34.907055 4934 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-nshph" Oct 31 04:26:35.385624 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:35.385595 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-kkqzx_283b8014-5ffd-4671-a28d-2975376d9cf9/cluster-monitoring-operator/0.log" Oct 31 04:26:40.458745 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:40.458714 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgq2q_f2cf121d-77e3-44a1-a62e-fd47b7504564/node-exporter/0.log" Oct 31 04:26:40.875718 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:40.875686 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgq2q_f2cf121d-77e3-44a1-a62e-fd47b7504564/kube-rbac-proxy/0.log" Oct 31 04:26:41.296862 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:41.296784 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgq2q_f2cf121d-77e3-44a1-a62e-fd47b7504564/init-textfile/0.log" Oct 31 04:26:49.164676 ip-10-0-138-238 kubenswrapper[4934]: E1031 04:26:49.164636 4934 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-z7s46" containerName="extractor" Oct 31 04:26:51.881005 ip-10-0-138-238 kubenswrapper[4934]: I1031 04:26:51.880961 4934 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-764498d796-fmmn5_8f03c061-d61a-4f36-9a03-4bdd462df099/kube-auth-proxy/0.log"