Nov 03 20:25:41.896152 ip-10-0-134-71 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 20:25:41.896161 ip-10-0-134-71 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 20:25:41.896168 ip-10-0-134-71 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 20:25:41.896414 ip-10-0-134-71 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 20:25:51.921024 ip-10-0-134-71 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 20:25:51.921039 ip-10-0-134-71 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c9ab5f11d4ac40d7b75809b6df88b8cd -- Nov 03 20:28:11.660450 ip-10-0-134-71 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:28:12.187798 ip-10-0-134-71 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:12.187798 ip-10-0-134-71 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:28:12.187798 ip-10-0-134-71 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:12.187798 ip-10-0-134-71 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:12.192281 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192186 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.192281 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192275 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.192281 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192280 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.192281 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192283 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.192281 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192286 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.192281 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192289 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.192281 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192292 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192295 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192297 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192300 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192303 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192306 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192309 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192312 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192315 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192317 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192320 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192323 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192325 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192328 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192331 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192335 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192340 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192343 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192346 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.192568 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192348 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192351 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192353 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192356 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192359 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192370 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192373 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192376 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192379 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192381 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192384 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192386 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192389 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192392 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192395 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192398 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192401 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192404 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192407 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192410 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.193018 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192413 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192415 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192418 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192421 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192423 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192427 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192430 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192433 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192435 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192438 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192441 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192444 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192446 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192449 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192451 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192456 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192459 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192462 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.193526 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192465 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192468 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192471 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192473 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192476 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192479 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192482 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192484 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192487 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192490 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192494 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192497 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192500 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192502 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192505 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192510 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192513 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192522 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.193970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192526 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192529 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192533 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192535 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192538 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192541 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192543 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192546 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192549 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192551 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192554 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192556 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192559 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192562 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192565 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192569 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192573 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192577 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192580 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.194429 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192583 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192588 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192591 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.192594 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193418 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193426 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193428 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193432 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193435 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193437 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193440 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193443 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193446 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193450 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193455 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193458 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193461 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193465 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193467 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.194871 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193470 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193473 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193476 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193478 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193481 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193484 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193486 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193489 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193492 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193495 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193497 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193500 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193503 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193505 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193508 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193511 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193514 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193516 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193519 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193521 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.195402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193524 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193527 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193529 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193532 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193535 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193538 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193541 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193543 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193546 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193549 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193552 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193554 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193557 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193559 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193561 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193564 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193567 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193569 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193571 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193575 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.195878 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193578 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193580 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193583 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193585 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193588 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193590 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193594 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193597 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193599 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193602 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193604 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193607 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193610 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193612 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193615 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193618 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193620 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193623 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193625 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.196380 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193628 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193632 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193637 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193640 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193642 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193645 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193648 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193651 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193653 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193659 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193661 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193664 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193667 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193670 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193672 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193675 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193677 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193680 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.196834 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193684 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193687 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193689 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193692 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193695 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193697 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193700 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193702 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193705 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193707 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193710 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193712 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.193715 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194228 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194234 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194237 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194240 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194243 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194245 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.197290 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194248 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194251 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194253 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194256 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194259 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194261 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194264 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194268 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194270 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194273 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194276 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194279 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194281 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194284 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194287 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194290 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194293 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194295 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194298 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194301 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.197833 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194303 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194306 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194310 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194314 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194317 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194320 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194322 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194325 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194328 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194331 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194334 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194337 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194340 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194343 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194345 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194348 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194351 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194353 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194356 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.198402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194359 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194362 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194365 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194368 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194370 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194373 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194376 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194380 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194384 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194387 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194390 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194393 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194396 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194398 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194401 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194403 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194406 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194408 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194411 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.198879 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194414 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194417 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194419 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194421 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194425 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194427 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194430 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194433 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194436 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194439 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194441 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194448 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194451 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194453 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194456 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194458 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194461 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194464 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194466 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.199343 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194471 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194474 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194476 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194479 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194482 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194484 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194487 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194490 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194493 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194496 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194498 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194501 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194503 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194506 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194508 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194511 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194514 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194516 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194520 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194522 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.199788 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194525 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194527 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.194530 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196845 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196859 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196870 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196874 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196879 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196882 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196886 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196891 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196894 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196898 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196901 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196905 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196908 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196911 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196914 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196917 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196920 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196923 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196927 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196930 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:28:12.200316 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196934 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196936 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196940 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196944 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196947 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196950 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196953 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196956 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196960 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196963 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196966 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196969 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196973 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196977 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196980 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196983 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196986 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196989 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196994 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.196997 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197000 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197003 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197006 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197011 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:28:12.200858 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197014 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197017 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197020 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197023 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197026 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197029 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197032 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197035 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197038 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197041 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197045 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197048 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197052 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197055 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197058 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197061 2578 flags.go:64] FLAG: --help="false" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197064 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197068 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197071 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197074 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197078 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197082 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197085 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:28:12.201463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197088 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197091 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197094 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197097 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197100 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197103 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197106 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197109 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197113 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197115 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197118 2578 flags.go:64] FLAG: --lock-file="" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197121 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197124 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197127 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197133 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197136 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197139 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197142 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197145 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197148 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197151 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197154 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197170 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197174 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197178 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 20:28:12.202004 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197181 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197184 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197188 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197191 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197194 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197197 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197200 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197207 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197210 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197213 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197216 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197219 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197222 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197225 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197228 2578 flags.go:64] FLAG: --port="10250" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197231 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197234 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a97411e7ded09ea5" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197237 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197240 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197243 2578 flags.go:64] FLAG: --register-node="true" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197246 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197250 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197253 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197256 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197259 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197263 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:28:12.202613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197266 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197269 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197272 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197278 2578 flags.go:64] FLAG: --runonce="false" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197281 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197284 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197287 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197290 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197292 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197296 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197300 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197303 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197306 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197309 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197312 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197315 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197318 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197321 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197325 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197331 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197334 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197338 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197342 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197345 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197348 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:28:12.203232 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197350 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197354 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197357 2578 flags.go:64] FLAG: --v="2" Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197361 2578 flags.go:64] FLAG: --version="false" Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197365 2578 flags.go:64] FLAG: --vmodule="" Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197369 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197373 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197516 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197521 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197525 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197532 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197535 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197538 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197541 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197544 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197547 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197550 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197553 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197555 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197558 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197561 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.203861 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197563 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197566 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197569 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197571 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197577 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197579 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197582 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197585 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197587 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197590 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197592 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197595 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197598 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197600 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197603 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197606 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197608 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197611 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197613 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197616 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.204446 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197618 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197622 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197625 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197628 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197630 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197633 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197636 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197638 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197642 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197647 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197650 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197653 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197655 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197658 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197660 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197663 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197667 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197669 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197672 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.204925 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197675 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197677 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197680 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197683 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197685 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197688 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197690 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197693 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197696 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197698 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197701 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197703 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197706 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197709 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197713 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197715 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197718 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197720 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197723 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197725 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.205402 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197728 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197730 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197733 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197737 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197739 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197742 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197744 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197747 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197750 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197754 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197756 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197759 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197761 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197764 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197767 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197769 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197772 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197775 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197777 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197780 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.205872 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197783 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197785 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197788 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197790 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197793 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197796 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197800 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197803 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197806 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197808 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197811 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197813 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197816 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.197842 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197879 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.206353 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197884 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197889 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197892 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197895 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197898 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197900 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197904 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197907 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197910 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197912 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197915 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197918 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197920 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197923 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197925 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197928 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197931 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197933 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197936 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197938 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.206717 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197941 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197943 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197946 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197949 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197952 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197955 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197958 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197960 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197963 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197966 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197969 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197971 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197974 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197976 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197979 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197982 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197985 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197988 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197991 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.207206 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197994 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.197998 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198002 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198005 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198008 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198011 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198013 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198016 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198018 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198021 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198023 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198026 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198029 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198031 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198034 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198036 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198039 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198043 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198045 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.207676 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198048 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198051 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198053 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198056 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198059 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198061 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198064 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198067 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198070 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198073 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198076 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198078 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198083 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198085 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198088 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198090 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198093 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198096 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198098 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198101 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.208123 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198103 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198106 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198109 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198111 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198114 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198116 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198119 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198121 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198124 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198127 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198131 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198133 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198136 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198138 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198141 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198144 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198146 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198150 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198152 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198166 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.208626 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198169 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198172 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198175 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198177 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198181 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198184 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.198187 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.205838 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.205856 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.205873 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.205879 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205939 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205944 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205947 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205950 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205953 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205956 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205959 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205961 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205964 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.209134 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205966 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205969 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205972 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205975 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205978 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205981 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205984 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205986 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205989 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205992 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205994 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205997 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.205999 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206002 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206004 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206007 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206010 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206013 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206015 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206018 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.209614 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206026 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206029 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206032 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206034 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206037 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206039 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206042 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206044 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206047 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206049 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206052 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206055 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206057 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206060 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206062 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206065 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206068 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206072 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206076 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.210080 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206079 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206082 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206085 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206088 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206091 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206093 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206096 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206098 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206101 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206104 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206106 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206109 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206112 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206114 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206123 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206126 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206128 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206130 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.210535 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206138 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206141 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206143 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206146 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206148 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206151 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206153 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206168 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206171 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206173 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206176 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206178 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206181 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206183 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206187 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206189 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206192 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206195 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206197 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.210990 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206199 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206202 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206205 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206208 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206210 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206213 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206215 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206218 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206226 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206229 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206232 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206234 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206237 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206240 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206242 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206245 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206247 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206250 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.211462 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206254 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.206282 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206469 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206474 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206477 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206481 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206484 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206487 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206489 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206492 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206494 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206497 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206500 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206502 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206505 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.211882 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206507 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206510 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206513 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206515 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206518 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206521 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206523 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206526 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206535 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206538 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206541 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206543 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206546 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206548 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206551 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206554 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206556 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206559 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206561 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206564 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.212254 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206567 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206569 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206572 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206574 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206577 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206579 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206582 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206584 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206587 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206589 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206592 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206594 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206597 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206600 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206602 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206605 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206607 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206610 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206612 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206615 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.212719 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206617 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206625 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206628 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206630 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206633 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206635 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206638 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206640 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206643 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206645 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206648 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206650 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206653 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206658 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206661 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206663 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206666 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206669 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206671 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.213243 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206674 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206676 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206685 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206687 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206690 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206692 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206695 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206698 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206700 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206703 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206705 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206708 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206710 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206713 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206721 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206724 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206727 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206730 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206732 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.213713 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206735 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206739 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206743 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206745 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206748 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206751 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206754 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206756 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206759 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206762 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206766 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206769 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206772 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206775 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.206798 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:12.214145 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206834 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206838 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206843 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206846 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206849 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206851 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206854 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206857 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206859 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206862 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206864 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206867 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206870 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206878 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206883 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206885 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206888 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206891 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206893 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206896 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:12.214512 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206898 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206901 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206903 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206906 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206908 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206911 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206913 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206916 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206919 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206921 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206924 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206926 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206929 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206931 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206935 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206938 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206940 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206943 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206946 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206948 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:12.214989 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206951 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206953 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206956 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206958 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206961 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206964 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206973 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206976 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206978 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206981 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206984 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206986 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206989 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206992 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206994 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.206997 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207000 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207002 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207005 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207007 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:12.215820 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207010 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207013 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207015 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207018 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207020 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207023 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207027 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207029 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207032 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207035 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207037 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207041 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207045 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207048 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207052 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207054 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207057 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207060 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207064 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:12.216659 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207074 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207077 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207081 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207084 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207087 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207089 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207092 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207095 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207098 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207101 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207104 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207106 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207109 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207112 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207115 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207117 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207120 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207123 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207125 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:12.217185 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207130 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207132 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207135 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207138 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207140 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207143 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207145 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:12.207148 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.207220 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.211413 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.212498 2578 server.go:1014] "Starting client certificate rotation" Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.212617 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.212644 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:12.217685 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.214907 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:12.219107 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.219069 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:12.229837 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.229817 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:28:12.234961 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.234944 2578 log.go:25] "Validated CRI v1 image API" Nov 03 20:28:12.235797 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.235771 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:28:12.241012 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.240988 2578 fs.go:135] Filesystem UUIDs: map[026769af-2496-4d2e-b354-bfb2edc1d8a8:/dev/nvme0n1p3 2b2391b5-24a0-402f-95c1-d0456070e434:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:28:12.241077 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.241010 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:28:12.246877 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.246759 2578 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:12.2451851 +0000 UTC m=+0.434581996 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098822 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2149c9c65fe5d121502aedd321e630 SystemUUID:ec2149c9-c65f-e5d1-2150-2aedd321e630 BootID:c9ab5f11-d4ac-40d7-b758-09b6df88b8cd Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d1:e5:2b:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d1:e5:2b:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:76:fd:67:8a:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:28:12.246877 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.246877 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:28:12.246982 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.246962 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:28:12.252760 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.252730 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:28:12.252900 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.252762 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-71.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:28:12.252952 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.252906 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:28:12.252952 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.252914 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:28:12.253005 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.252964 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:12.253005 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.252984 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:12.254190 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.254181 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:28:12.258502 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.258490 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:28:12.258628 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.258619 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:28:12.259385 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.259376 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:28:12.259417 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.259389 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:28:12.259417 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.259401 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:28:12.259417 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.259410 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:28:12.259504 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.259420 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:28:12.259850 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.259834 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:12.262607 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.262577 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:28:12.269421 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.269406 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:12.270315 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270302 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:28:12.270363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270332 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:28:12.270479 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270470 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:28:12.270510 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270484 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:28:12.270510 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270491 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:28:12.270510 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270496 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:28:12.270510 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270502 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:28:12.270510 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.270507 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:28:12.271826 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.271813 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:28:12.271862 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.271826 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:28:12.271862 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.271834 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:28:12.271862 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.271841 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:28:12.271862 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.271850 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:28:12.271862 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.271859 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:28:12.273132 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.273122 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:28:12.273132 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.273132 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:28:12.277230 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.277203 2578 server.go:1263] "Started kubelet" Nov 03 20:28:12.277650 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.277620 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:28:12.277814 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.277601 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:28:12.277957 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.277905 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:28:12.278036 ip-10-0-134-71 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:28:12.279280 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.279259 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:28:12.280896 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.280881 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:28:12.284373 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.284356 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:28:12.287947 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.287929 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:12.289176 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.289144 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:28:12.289363 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.289343 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" Nov 03 20:28:12.289992 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.289972 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:28:12.289992 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.289974 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:28:12.290113 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.290004 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:28:12.290185 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.290119 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:28:12.290185 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.290125 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:28:12.290982 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.290964 2578 factory.go:55] Registering systemd factory Nov 03 20:28:12.290982 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.290983 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:28:12.291288 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.291273 2578 factory.go:153] Registering CRI-O factory Nov 03 20:28:12.291333 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.291293 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 20:28:12.291362 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.291346 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:28:12.291404 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.291364 2578 factory.go:103] Registering Raw factory Nov 03 20:28:12.291404 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.291383 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 20:28:12.292332 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.292316 2578 manager.go:319] Starting recovery of all containers Nov 03 20:28:12.301748 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.301734 2578 manager.go:324] Recovery completed Nov 03 20:28:12.305714 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.305700 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:12.307434 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.307417 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:12.307495 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.307450 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:12.307495 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.307461 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:12.308186 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.308149 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:28:12.308265 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.308187 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:28:12.308265 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.308213 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:28:12.308368 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.308280 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:12.309449 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.309428 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:12.309526 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.309468 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:12.309526 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.309484 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:12.310491 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.310476 2578 policy_none.go:50] "Start" Nov 03 20:28:12.310548 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.310500 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:28:12.310548 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.310519 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:28:12.312497 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.312487 2578 policy_none.go:44] "Start" Nov 03 20:28:12.354731 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.354701 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:28:12.356938 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.356924 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 20:28:12.357011 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.356949 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:28:12.357011 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.356957 2578 server.go:87] "Starting device plugin registration server" Nov 03 20:28:12.357211 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.357198 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:28:12.357275 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.357213 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:28:12.357310 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.357277 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:28:12.357369 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.357356 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:28:12.357419 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.357370 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:28:12.357983 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.357829 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:28:12.358055 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.358020 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" Nov 03 20:28:12.421333 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.421305 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-71.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:28:12.421333 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.421308 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-71.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:28:12.423347 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.421305 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-71.us-west-2.compute.internal.18e4a2ed7068876c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-71.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-71.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:12.2771475 +0000 UTC m=+0.466544397,LastTimestamp:2026-11-03 20:28:12.2771475 +0000 UTC m=+0.466544397,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-71.us-west-2.compute.internal,}" Nov 03 20:28:12.436799 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.436764 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:28:12.437970 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.437924 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:28:12.437970 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.437944 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:28:12.437970 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.437966 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:28:12.438175 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.437996 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:28:12.457993 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.457973 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:12.458810 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.458794 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:12.458887 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.458823 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:12.458887 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.458834 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:12.458887 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.458858 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.496997 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.496970 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:28:12.497107 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.497038 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cbvrd" Nov 03 20:28:12.538646 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.538621 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal"] Nov 03 20:28:12.538737 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.538685 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:12.539584 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.539569 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:12.539641 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.539602 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:12.539641 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.539615 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:12.541821 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.541809 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:12.541976 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.541961 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.542020 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.541995 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:12.542538 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.542511 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:12.542603 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.542546 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:12.542603 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.542556 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:12.542603 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.542564 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:12.542603 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.542590 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:12.542603 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.542601 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:12.544651 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.544632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.544738 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.544656 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:12.545354 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.545340 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:12.545422 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.545360 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:12.545422 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.545371 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:12.577859 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.577840 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.580363 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.580344 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.580581 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.580568 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:12.582376 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.582363 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:12.593112 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.593095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.593180 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.593119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.593180 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.593139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b295d4bbd28e0091df21418ae66e44d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"b295d4bbd28e0091df21418ae66e44d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.682227 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.682204 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:12.685896 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.685657 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.685896 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:12.685691 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-71.us-west-2.compute.internal\": node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" Nov 03 20:28:12.686061 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.685994 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:12.689453 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.689397 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.693794 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.693777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.693857 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.693801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.693857 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.693818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b295d4bbd28e0091df21418ae66e44d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"b295d4bbd28e0091df21418ae66e44d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.693931 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.693861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b295d4bbd28e0091df21418ae66e44d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"b295d4bbd28e0091df21418ae66e44d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.693931 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.693882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.693931 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.693893 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.750729 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.750710 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:12.750813 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.750802 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.813592 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.813569 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:12.882719 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.882695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:12.885025 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:12.885008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:13.212514 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.212492 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:28:13.213092 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.212625 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:28:13.213092 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.212660 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:28:13.213092 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.212658 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:28:13.213092 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.212692 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a6eb617b3f8214fa495946204f12b38d-1c21b587483bc72e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-71.us-west-2.compute.internal.18e4a2ed7236fe7e\": read tcp 10.0.134.71:48862->3.149.15.198:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-71.us-west-2.compute.internal.18e4a2ed7236fe7e default 5241 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-71.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-71.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-71.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:12 +0000 UTC,LastTimestamp:2026-11-03 20:28:12.458827659 +0000 UTC m=+0.648224555,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-71.us-west-2.compute.internal,}" Nov 03 20:28:13.262186 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.262155 2578 apiserver.go:52] "Watching apiserver" Nov 03 20:28:13.289315 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.289291 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:13.502484 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.502409 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:12 +0000 UTC" deadline="2028-07-09 17:15:58.739848413 +0000 UTC" Nov 03 20:28:13.502484 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.502445 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14732h47m45.237407188s" Nov 03 20:28:13.524349 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.524326 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:28:13.524670 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.524650 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-gcz42","kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal","openshift-dns/node-resolver-4wjxm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-xqvnn","kube-system/konnectivity-agent-hc7pp","openshift-cluster-node-tuning-operator/tuned-92trd","openshift-ovn-kubernetes/ovnkube-node-xx8t5","kube-system/global-pull-secret-syncer-xfqt9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc","openshift-network-diagnostics/network-check-target-7v5wf","openshift-image-registry/node-ca-dg8z2","openshift-multus/multus-additional-cni-plugins-jr82s","openshift-multus/multus-wwvxj"] Nov 03 20:28:13.527331 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.527311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.529555 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.529537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:13.529643 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.529593 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:13.531737 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.531718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.533954 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.533935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.536090 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.536074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.538310 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.538291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.540675 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.540661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.543067 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.543050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.545728 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.545531 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.545728 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.545610 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:13.548261 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.548241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:13.548352 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.548332 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:13.550603 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.550586 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.554611 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.554590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.580585 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.580547 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:28:13.580585 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.580554 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:28:13.580585 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.580571 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6582\"" Nov 03 20:28:13.580744 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.580632 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:28:13.584335 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.584311 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:28:13.584418 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.584404 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:28:13.584480 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.584467 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mlgd9\"" Nov 03 20:28:13.586607 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.586595 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:28:13.586683 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.586629 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:28:13.586683 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.586653 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9qmc\"" Nov 03 20:28:13.586683 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.586670 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:28:13.586794 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.586685 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:28:13.588325 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.588308 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:13.588551 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.588538 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8pktm\"" Nov 03 20:28:13.588628 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.588615 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:13.590777 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.590759 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:28:13.590890 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.590816 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:28:13.591022 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.591003 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d54s8\"" Nov 03 20:28:13.591340 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.591326 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:28:13.593113 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.593096 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:13.593261 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.593248 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:13.597414 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a652af6-69fa-467b-abfe-bff66e3afbde-agent-certs\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.597497 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcpt6\" (UniqueName: \"kubernetes.io/projected/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kube-api-access-mcpt6\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.597497 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-sys\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.597497 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.597497 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11a94da7-4353-4baf-9cd9-e585ff91ac82-serviceca\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.597633 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-slash\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.597633 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-hosts-file\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.597633 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-systemd\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.597633 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fa46f4-4f1e-4092-a839-1dd075263b50-host-slash\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.597753 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-kubelet\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.597753 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-bin\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.597753 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.597753 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-modprobe-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.597871 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a652af6-69fa-467b-abfe-bff66e3afbde-konnectivity-ca\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.597871 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-dbus\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.597871 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-conf-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.597871 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-daemon-config\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.597871 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-cnibin\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.597871 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598083 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-multus\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.598083 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-log-socket\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598083 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-run\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598083 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-host\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598083 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.597998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.598083 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-config\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598083 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.598312 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.598312 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9fa46f4-4f1e-4092-a839-1dd075263b50-iptables-alerter-script\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.598312 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w6wb\" (UniqueName: \"kubernetes.io/projected/13dda6c1-f004-46b6-928c-47d6de1d58f9-kube-api-access-6w6wb\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.598312 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-script-lib\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysconfig\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-k8s-cni-cncf-io\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a94da7-4353-4baf-9cd9-e585ff91ac82-host\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-env-overrides\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovn-node-metrics-cert\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkqf5\" (UniqueName: \"kubernetes.io/projected/0568938d-c9e2-4e37-b768-cdeddaacf33e-kube-api-access-xkqf5\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598452 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-var-lib-kubelet\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-tuned\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-cni-binary-copy\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kph6l\" (UniqueName: \"kubernetes.io/projected/e9fa46f4-4f1e-4092-a839-1dd075263b50-kube-api-access-kph6l\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-conf\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-tmp\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-lib-modules\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-tmp-dir\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-bin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-netns\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.598803 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-node-log\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4gs\" (UniqueName: \"kubernetes.io/projected/e55a9e03-4858-42b2-a27a-a9533e404588-kube-api-access-7b4gs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-system-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-ovn\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmnk\" (UniqueName: \"kubernetes.io/projected/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-kube-api-access-klmnk\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.598991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-kubernetes\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-cnibin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-kubelet\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-multus-certs\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-systemd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-hostroot\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-system-cni-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.599336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lnc\" (UniqueName: \"kubernetes.io/projected/11a94da7-4353-4baf-9cd9-e585ff91ac82-kube-api-access-d5lnc\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-etc-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-netd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-netns\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9g74\" (UniqueName: \"kubernetes.io/projected/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-kube-api-access-z9g74\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-kubelet-config\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-socket-dir-parent\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-etc-kubernetes\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4h2g\" (UniqueName: \"kubernetes.io/projected/9d5cb641-099e-4366-9d79-85d72a0e1a90-kube-api-access-g4h2g\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-systemd-units\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-var-lib-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.599841 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.599701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.628311 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.628296 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jb4cj\"" Nov 03 20:28:13.628406 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.628315 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:28:13.629845 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.629830 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:28:13.629927 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.629880 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kb76x" Nov 03 20:28:13.689347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689316 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:28:13.689491 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689418 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:28:13.689572 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689532 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:28:13.689620 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689578 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxhx8\"" Nov 03 20:28:13.689677 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689631 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:28:13.689677 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689659 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:28:13.689677 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689665 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:28:13.689677 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689666 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:28:13.689874 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689681 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:28:13.689874 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689684 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:28:13.689874 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689666 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w48hg\"" Nov 03 20:28:13.689874 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689720 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:28:13.689874 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689836 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:28:13.689874 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.689873 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6vtd\"" Nov 03 20:28:13.700510 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-run\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.700510 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-host\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.700698 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.700698 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-config\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.700698 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-host\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.700698 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.700698 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700699 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700699 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-run\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9fa46f4-4f1e-4092-a839-1dd075263b50-iptables-alerter-script\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w6wb\" (UniqueName: \"kubernetes.io/projected/13dda6c1-f004-46b6-928c-47d6de1d58f9-kube-api-access-6w6wb\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700771 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-script-lib\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysconfig\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-k8s-cni-cncf-io\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a94da7-4353-4baf-9cd9-e585ff91ac82-host\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.700923 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-env-overrides\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a94da7-4353-4baf-9cd9-e585ff91ac82-host\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysconfig\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovn-node-metrics-cert\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.700987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-k8s-cni-cncf-io\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqf5\" (UniqueName: \"kubernetes.io/projected/0568938d-c9e2-4e37-b768-cdeddaacf33e-kube-api-access-xkqf5\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-var-lib-kubelet\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-tuned\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-cni-binary-copy\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph6l\" (UniqueName: \"kubernetes.io/projected/e9fa46f4-4f1e-4092-a839-1dd075263b50-kube-api-access-kph6l\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-conf\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-tmp\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-lib-modules\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-tmp-dir\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-bin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.701530 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-var-lib-kubelet\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-netns\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701375 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-node-log\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-env-overrides\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-config\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4gs\" (UniqueName: \"kubernetes.io/projected/e55a9e03-4858-42b2-a27a-a9533e404588-kube-api-access-7b4gs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-script-lib\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9fa46f4-4f1e-4092-a839-1dd075263b50-iptables-alerter-script\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-system-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701504 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-bin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-ovn\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-system-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmnk\" (UniqueName: \"kubernetes.io/projected/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-kube-api-access-klmnk\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-ovn\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.702430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-netns\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-kubernetes\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-conf\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-cnibin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-kubernetes\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-kubelet\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701726 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-node-log\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-multus-certs\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-systemd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701832 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-hostroot\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-system-cni-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-hostroot\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lnc\" (UniqueName: \"kubernetes.io/projected/11a94da7-4353-4baf-9cd9-e585ff91ac82-kube-api-access-d5lnc\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.703363 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-cnibin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-etc-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701957 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-kubelet\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-netd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-netd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701997 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-systemd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-etc-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.702110 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-lib-modules\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.701991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-netns\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-multus-certs\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-netns\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.702317 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:14.202281406 +0000 UTC m=+2.391678306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-cni-binary-copy\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.703996 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702365 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-system-cni-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g74\" (UniqueName: \"kubernetes.io/projected/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-kube-api-access-z9g74\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-kubelet-config\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-tmp-dir\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-kubelet-config\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-socket-dir-parent\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-etc-kubernetes\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4h2g\" (UniqueName: \"kubernetes.io/projected/9d5cb641-099e-4366-9d79-85d72a0e1a90-kube-api-access-g4h2g\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-systemd-units\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-var-lib-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-etc-kubernetes\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a652af6-69fa-467b-abfe-bff66e3afbde-agent-certs\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-socket-dir-parent\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcpt6\" (UniqueName: \"kubernetes.io/projected/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kube-api-access-mcpt6\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.704743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-sys\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-systemd-units\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11a94da7-4353-4baf-9cd9-e585ff91ac82-serviceca\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-slash\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-hosts-file\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-slash\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-systemd\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-var-lib-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fa46f4-4f1e-4092-a839-1dd075263b50-host-slash\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-kubelet\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702961 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-bin\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.703028 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-modprobe-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.705681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a652af6-69fa-467b-abfe-bff66e3afbde-konnectivity-ca\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-dbus\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.703143 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret podName:8eddcaf9-30b4-44e8-bc77-24b08f234497 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:14.203119809 +0000 UTC m=+2.392516709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret") pod "global-pull-secret-syncer-xfqt9" (UID: "8eddcaf9-30b4-44e8-bc77-24b08f234497") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703171 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fa46f4-4f1e-4092-a839-1dd075263b50-host-slash\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-conf-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-conf-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-daemon-config\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-dbus\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-cnibin\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-hosts-file\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-multus\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11a94da7-4353-4baf-9cd9-e585ff91ac82-serviceca\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-log-socket\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-systemd\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.706347 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.702900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-sys\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-multus\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-log-socket\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-bin\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-cnibin\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-kubelet\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703616 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-modprobe-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.703651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.704713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a652af6-69fa-467b-abfe-bff66e3afbde-konnectivity-ca\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.704920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-daemon-config\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.705737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-tuned\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.705902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovn-node-metrics-cert\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.705969 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a652af6-69fa-467b-abfe-bff66e3afbde-agent-certs\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.706832 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.706152 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-tmp\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.760994 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.760907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqf5\" (UniqueName: \"kubernetes.io/projected/0568938d-c9e2-4e37-b768-cdeddaacf33e-kube-api-access-xkqf5\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:13.761136 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.761118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g74\" (UniqueName: \"kubernetes.io/projected/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-kube-api-access-z9g74\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.761198 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.761138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4gs\" (UniqueName: \"kubernetes.io/projected/e55a9e03-4858-42b2-a27a-a9533e404588-kube-api-access-7b4gs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:13.762389 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.762314 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:13.762389 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.762340 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:13.762389 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.762355 2578 projected.go:196] Error preparing data for projected volume kube-api-access-69j74 for pod openshift-network-diagnostics/network-check-target-7v5wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:13.762654 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:13.762429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74 podName:662d94e5-925a-4454-af65-8f767b48f365 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:14.26240751 +0000 UTC m=+2.451804410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-69j74" (UniqueName: "kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74") pod "network-check-target-7v5wf" (UID: "662d94e5-925a-4454-af65-8f767b48f365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:13.764873 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.764815 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcpt6\" (UniqueName: \"kubernetes.io/projected/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kube-api-access-mcpt6\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.764873 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.764829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w6wb\" (UniqueName: \"kubernetes.io/projected/13dda6c1-f004-46b6-928c-47d6de1d58f9-kube-api-access-6w6wb\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.765280 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.765256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph6l\" (UniqueName: \"kubernetes.io/projected/e9fa46f4-4f1e-4092-a839-1dd075263b50-kube-api-access-kph6l\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.765368 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.765327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lnc\" (UniqueName: \"kubernetes.io/projected/11a94da7-4353-4baf-9cd9-e585ff91ac82-kube-api-access-d5lnc\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.765368 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.765340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4h2g\" (UniqueName: \"kubernetes.io/projected/9d5cb641-099e-4366-9d79-85d72a0e1a90-kube-api-access-g4h2g\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.765492 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.765475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmnk\" (UniqueName: \"kubernetes.io/projected/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-kube-api-access-klmnk\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.866093 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.866060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:13.880599 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.880574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:13.896958 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.896934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wwvxj" Nov 03 20:28:13.916139 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.916122 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:13.916216 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.916199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:13.939325 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.939303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:13.969566 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.969543 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:13.971872 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.971854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:13.992037 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:13.992018 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:14.104934 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.104902 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:14.118037 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.118008 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb295d4bbd28e0091df21418ae66e44d9.slice/crio-f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b WatchSource:0}: Error finding container f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b: Status 404 returned error can't find the container with id f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b Nov 03 20:28:14.125523 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.125502 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:28:14.128851 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.128829 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c24ecfbcdfa9586bb1da6fb1e34a7c.slice/crio-eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133 WatchSource:0}: Error finding container eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133: Status 404 returned error can't find the container with id eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133 Nov 03 20:28:14.207050 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.207022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:14.207150 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.207066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:14.207217 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.207195 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:14.207261 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.207235 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret podName:8eddcaf9-30b4-44e8-bc77-24b08f234497 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:15.207223176 +0000 UTC m=+3.396620060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret") pod "global-pull-secret-syncer-xfqt9" (UID: "8eddcaf9-30b4-44e8-bc77-24b08f234497") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:14.207319 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.207265 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:14.207319 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.207311 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:15.207298964 +0000 UTC m=+3.396695868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:14.307741 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.307711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:14.308273 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.307835 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:14.308273 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.307854 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:14.308273 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.307863 2578 projected.go:196] Error preparing data for projected volume kube-api-access-69j74 for pod openshift-network-diagnostics/network-check-target-7v5wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:14.308273 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:14.307904 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74 podName:662d94e5-925a-4454-af65-8f767b48f365 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:15.307891868 +0000 UTC m=+3.497288768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-69j74" (UniqueName: "kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74") pod "network-check-target-7v5wf" (UID: "662d94e5-925a-4454-af65-8f767b48f365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:14.441857 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.441778 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"b295d4bbd28e0091df21418ae66e44d9","Type":"ContainerStarted","Data":"f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b"} Nov 03 20:28:14.442838 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.442818 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"65c24ecfbcdfa9586bb1da6fb1e34a7c","Type":"ContainerStarted","Data":"eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133"} Nov 03 20:28:14.566215 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.566190 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:14.616140 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.616116 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:14.632703 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.632657 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:13 +0000 UTC" deadline="2028-04-28 12:10:21.189601258 +0000 UTC" Nov 03 20:28:14.632703 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:14.632696 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12999h42m6.556909402s" Nov 03 20:28:14.836994 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.836967 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228f5775_1d78_4310_a6d4_fccb3f18a4f4.slice/crio-5f0b001be9f8130d7f4d6fd8eea51504a326ee41258e2dd23539a960934cd85e WatchSource:0}: Error finding container 5f0b001be9f8130d7f4d6fd8eea51504a326ee41258e2dd23539a960934cd85e: Status 404 returned error can't find the container with id 5f0b001be9f8130d7f4d6fd8eea51504a326ee41258e2dd23539a960934cd85e Nov 03 20:28:14.869880 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.869847 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5cb641_099e_4366_9d79_85d72a0e1a90.slice/crio-0ed7e4eb9858d00756e506026c660a1d367c159e38c375d1a78fe709a4ba8a68 WatchSource:0}: Error finding container 0ed7e4eb9858d00756e506026c660a1d367c159e38c375d1a78fe709a4ba8a68: Status 404 returned error can't find the container with id 0ed7e4eb9858d00756e506026c660a1d367c159e38c375d1a78fe709a4ba8a68 Nov 03 20:28:14.886970 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.886946 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba768925_3dbc_4dcb_a35c_54e20f5bf36b.slice/crio-9ea52363775ce260552dc22e788fd389489806ae5ce0b56fd518b2d66a26b3bc WatchSource:0}: Error finding container 9ea52363775ce260552dc22e788fd389489806ae5ce0b56fd518b2d66a26b3bc: Status 404 returned error can't find the container with id 9ea52363775ce260552dc22e788fd389489806ae5ce0b56fd518b2d66a26b3bc Nov 03 20:28:14.904013 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.903990 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a94da7_4353_4baf_9cd9_e585ff91ac82.slice/crio-cad8a281346db8086044899f04ea993d595166fdae54ba4b72a52b998a83486a WatchSource:0}: Error finding container cad8a281346db8086044899f04ea993d595166fdae54ba4b72a52b998a83486a: Status 404 returned error can't find the container with id cad8a281346db8086044899f04ea993d595166fdae54ba4b72a52b998a83486a Nov 03 20:28:14.921817 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.921791 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa839ba6_7b29_41c8_bd50_a7cd5b1ac139.slice/crio-0a369a7f73a85eef1737426f905fc9b1695ba79250be6e1db369d4c08e80d141 WatchSource:0}: Error finding container 0a369a7f73a85eef1737426f905fc9b1695ba79250be6e1db369d4c08e80d141: Status 404 returned error can't find the container with id 0a369a7f73a85eef1737426f905fc9b1695ba79250be6e1db369d4c08e80d141 Nov 03 20:28:14.957767 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.957741 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fa46f4_4f1e_4092_a839_1dd075263b50.slice/crio-b5765c4baad6b0b8d0559cf221c7257cce01a022f8c455c4933c30594bc56af6 WatchSource:0}: Error finding container b5765c4baad6b0b8d0559cf221c7257cce01a022f8c455c4933c30594bc56af6: Status 404 returned error can't find the container with id b5765c4baad6b0b8d0559cf221c7257cce01a022f8c455c4933c30594bc56af6 Nov 03 20:28:14.958434 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.958410 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a652af6_69fa_467b_abfe_bff66e3afbde.slice/crio-0a12e8b1edba5656cb5309eb74313ebe671e47c7871aa043e366a985ca8bf157 WatchSource:0}: Error finding container 0a12e8b1edba5656cb5309eb74313ebe671e47c7871aa043e366a985ca8bf157: Status 404 returned error can't find the container with id 0a12e8b1edba5656cb5309eb74313ebe671e47c7871aa043e366a985ca8bf157 Nov 03 20:28:14.999276 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:14.999253 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dda6c1_f004_46b6_928c_47d6de1d58f9.slice/crio-12d6fa1c8cce7fb251045fd4e88e38b30b084792c3c99018abbbec3602f484c0 WatchSource:0}: Error finding container 12d6fa1c8cce7fb251045fd4e88e38b30b084792c3c99018abbbec3602f484c0: Status 404 returned error can't find the container with id 12d6fa1c8cce7fb251045fd4e88e38b30b084792c3c99018abbbec3602f484c0 Nov 03 20:28:15.000473 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:15.000454 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0568938d_c9e2_4e37_b768_cdeddaacf33e.slice/crio-41f236f00d4432250b764f7c2e71d99483e2036dab2684cbb0a9ecc4de46c278 WatchSource:0}: Error finding container 41f236f00d4432250b764f7c2e71d99483e2036dab2684cbb0a9ecc4de46c278: Status 404 returned error can't find the container with id 41f236f00d4432250b764f7c2e71d99483e2036dab2684cbb0a9ecc4de46c278 Nov 03 20:28:15.214285 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.214253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:15.214420 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.214381 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:15.214492 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.214412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:15.214492 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.214445 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:17.214425896 +0000 UTC m=+5.403823017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:15.214608 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.214531 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:15.214608 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.214574 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret podName:8eddcaf9-30b4-44e8-bc77-24b08f234497 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:17.214560898 +0000 UTC m=+5.403957818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret") pod "global-pull-secret-syncer-xfqt9" (UID: "8eddcaf9-30b4-44e8-bc77-24b08f234497") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:15.315381 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.315297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:15.315938 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.315479 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:15.315938 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.315499 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:15.315938 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.315509 2578 projected.go:196] Error preparing data for projected volume kube-api-access-69j74 for pod openshift-network-diagnostics/network-check-target-7v5wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:15.315938 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.315564 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74 podName:662d94e5-925a-4454-af65-8f767b48f365 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:17.315549187 +0000 UTC m=+5.504946086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-69j74" (UniqueName: "kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74") pod "network-check-target-7v5wf" (UID: "662d94e5-925a-4454-af65-8f767b48f365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:15.438429 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.438248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:15.438429 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.438367 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:15.438429 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.438381 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:15.438691 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.438478 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:15.438691 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.438525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:15.438691 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:15.438667 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:15.446033 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.446004 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dg8z2" event={"ID":"11a94da7-4353-4baf-9cd9-e585ff91ac82","Type":"ContainerStarted","Data":"cad8a281346db8086044899f04ea993d595166fdae54ba4b72a52b998a83486a"} Nov 03 20:28:15.447613 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.447587 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-92trd" event={"ID":"ba768925-3dbc-4dcb-a35c-54e20f5bf36b","Type":"ContainerStarted","Data":"9ea52363775ce260552dc22e788fd389489806ae5ce0b56fd518b2d66a26b3bc"} Nov 03 20:28:15.449039 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.449009 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" event={"ID":"228f5775-1d78-4310-a6d4-fccb3f18a4f4","Type":"ContainerStarted","Data":"5f0b001be9f8130d7f4d6fd8eea51504a326ee41258e2dd23539a960934cd85e"} Nov 03 20:28:15.450521 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.450496 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"41f236f00d4432250b764f7c2e71d99483e2036dab2684cbb0a9ecc4de46c278"} Nov 03 20:28:15.451863 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.451837 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gcz42" event={"ID":"e9fa46f4-4f1e-4092-a839-1dd075263b50","Type":"ContainerStarted","Data":"b5765c4baad6b0b8d0559cf221c7257cce01a022f8c455c4933c30594bc56af6"} Nov 03 20:28:15.453372 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.453346 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wjxm" event={"ID":"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139","Type":"ContainerStarted","Data":"0a369a7f73a85eef1737426f905fc9b1695ba79250be6e1db369d4c08e80d141"} Nov 03 20:28:15.454631 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.454606 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wwvxj" event={"ID":"9d5cb641-099e-4366-9d79-85d72a0e1a90","Type":"ContainerStarted","Data":"0ed7e4eb9858d00756e506026c660a1d367c159e38c375d1a78fe709a4ba8a68"} Nov 03 20:28:15.456104 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.456081 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerStarted","Data":"12d6fa1c8cce7fb251045fd4e88e38b30b084792c3c99018abbbec3602f484c0"} Nov 03 20:28:15.460761 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.460643 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hc7pp" event={"ID":"9a652af6-69fa-467b-abfe-bff66e3afbde","Type":"ContainerStarted","Data":"0a12e8b1edba5656cb5309eb74313ebe671e47c7871aa043e366a985ca8bf157"} Nov 03 20:28:15.633633 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.633530 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:13 +0000 UTC" deadline="2028-08-13 10:18:08.67658743 +0000 UTC" Nov 03 20:28:15.633633 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:15.633586 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15565h49m53.043005732s" Nov 03 20:28:17.239387 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:17.238876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:17.239387 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:17.238946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:17.239387 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.239087 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:17.239387 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.239149 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret podName:8eddcaf9-30b4-44e8-bc77-24b08f234497 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:21.239130443 +0000 UTC m=+9.428527330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret") pod "global-pull-secret-syncer-xfqt9" (UID: "8eddcaf9-30b4-44e8-bc77-24b08f234497") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:17.239387 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.239238 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:17.239387 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.239278 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:21.239265653 +0000 UTC m=+9.428662552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:17.339794 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:17.339759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:17.339980 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.339910 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:17.339980 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.339927 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:17.339980 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.339937 2578 projected.go:196] Error preparing data for projected volume kube-api-access-69j74 for pod openshift-network-diagnostics/network-check-target-7v5wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:17.339980 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.339980 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74 podName:662d94e5-925a-4454-af65-8f767b48f365 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:21.33996781 +0000 UTC m=+9.529364709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-69j74" (UniqueName: "kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74") pod "network-check-target-7v5wf" (UID: "662d94e5-925a-4454-af65-8f767b48f365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:17.438591 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:17.438555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:17.438822 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.438708 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:17.438822 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:17.438790 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:17.438977 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.438894 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:17.438977 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:17.438962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:17.439094 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:17.439059 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:18.493683 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:18.493471 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"b295d4bbd28e0091df21418ae66e44d9","Type":"ContainerStarted","Data":"870cde1061065317517a9fb74f059bd9ea7fc6c0c0303eaee0d4e6831db95d31"} Nov 03 20:28:19.439177 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.439133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:19.439460 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:19.439284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:19.439460 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.439349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:19.439460 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:19.439414 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:19.439460 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.439459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:19.439681 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:19.439582 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:19.512228 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.512194 2578 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="2c7c5e0f4dd1b66c9aaf5abefbf1dddc3aad7b91f6b7513583138d15e9918095" exitCode=0 Nov 03 20:28:19.512713 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.512312 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerDied","Data":"2c7c5e0f4dd1b66c9aaf5abefbf1dddc3aad7b91f6b7513583138d15e9918095"} Nov 03 20:28:19.520566 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.519516 2578 generic.go:356] "Generic (PLEG): container finished" podID="65c24ecfbcdfa9586bb1da6fb1e34a7c" containerID="0a86cabacfe43ca317860846cb27d734061f0f05344b77368c68076e125a8d88" exitCode=0 Nov 03 20:28:19.520566 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.520422 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"65c24ecfbcdfa9586bb1da6fb1e34a7c","Type":"ContainerDied","Data":"0a86cabacfe43ca317860846cb27d734061f0f05344b77368c68076e125a8d88"} Nov 03 20:28:19.718336 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:19.718275 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" podStartSLOduration=6.718255783 podStartE2EDuration="6.718255783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:18.599555571 +0000 UTC m=+6.788952475" watchObservedRunningTime="2026-11-03 20:28:19.718255783 +0000 UTC m=+7.907652689" Nov 03 20:28:20.527110 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:20.527073 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hc7pp" event={"ID":"9a652af6-69fa-467b-abfe-bff66e3afbde","Type":"ContainerStarted","Data":"3aeec7fb131a6c048a3b96db46651aa61a3e3a47cbccba87793d7f8179761994"} Nov 03 20:28:20.530778 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:20.530725 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"65c24ecfbcdfa9586bb1da6fb1e34a7c","Type":"ContainerStarted","Data":"05255c862fc64996d4a7a02ec2b1e690a39e89edd6e673117ff3759dcab6fbfe"} Nov 03 20:28:20.633169 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:20.633095 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hc7pp" podStartSLOduration=3.679450745 podStartE2EDuration="8.633074652s" totalImagesPullingTime="4.953623907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:20.632629572 +0000 UTC m=+8.822026476" watchObservedRunningTime="2026-11-03 20:28:20.633074652 +0000 UTC m=+8.822471558" Nov 03 20:28:20.738430 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:20.738381 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" podStartSLOduration=7.738367473 podStartE2EDuration="7.738367473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:20.737910844 +0000 UTC m=+8.927307750" watchObservedRunningTime="2026-11-03 20:28:20.738367473 +0000 UTC m=+8.927764378" Nov 03 20:28:21.271146 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.271116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:21.271280 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.271192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:21.271342 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.271315 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:21.271397 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.271387 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret podName:8eddcaf9-30b4-44e8-bc77-24b08f234497 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:29.271356532 +0000 UTC m=+17.460753414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret") pod "global-pull-secret-syncer-xfqt9" (UID: "8eddcaf9-30b4-44e8-bc77-24b08f234497") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:21.271460 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.271393 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:21.271460 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.271457 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:29.271440245 +0000 UTC m=+17.460837142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:21.372577 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.372500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:21.372714 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.372620 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:21.372714 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.372643 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:21.372714 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.372655 2578 projected.go:196] Error preparing data for projected volume kube-api-access-69j74 for pod openshift-network-diagnostics/network-check-target-7v5wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:21.372714 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.372702 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74 podName:662d94e5-925a-4454-af65-8f767b48f365 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:29.372689305 +0000 UTC m=+17.562086204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-69j74" (UniqueName: "kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74") pod "network-check-target-7v5wf" (UID: "662d94e5-925a-4454-af65-8f767b48f365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:21.438980 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.438938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:21.439075 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.439022 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:21.439130 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.439092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:21.439195 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.439177 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:21.439257 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.439223 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:21.439303 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:21.439284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:21.537325 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.537242 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dg8z2" event={"ID":"11a94da7-4353-4baf-9cd9-e585ff91ac82","Type":"ContainerStarted","Data":"a00d5b8b01c9171613b3312ba2847623f75695e8e9db6956b27db5cd9def9a90"} Nov 03 20:28:21.542681 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.542657 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" event={"ID":"228f5775-1d78-4310-a6d4-fccb3f18a4f4","Type":"ContainerStarted","Data":"e9b807e6f6e9d61f1d58bc690317715cdcdeda6ce786e62092013cc1346b2fc4"} Nov 03 20:28:21.645514 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:21.645470 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dg8z2" podStartSLOduration=2.620884188 podStartE2EDuration="8.645456234s" totalImagesPullingTime="6.024572046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:21.644569423 +0000 UTC m=+9.833966332" watchObservedRunningTime="2026-11-03 20:28:21.645456234 +0000 UTC m=+9.834853138" Nov 03 20:28:23.439015 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:23.438598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:23.439015 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:23.438706 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:23.439015 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:23.438771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:23.439015 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:23.438844 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:23.439015 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:23.438902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:23.439015 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:23.438977 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:23.699587 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:23.699550 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:28:24.363591 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.363298 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:28:23.699582427Z","UUID":"bdc5e9d6-3096-43dc-9364-f8d57d19ac13","Handler":null,"Name":"","Endpoint":""} Nov 03 20:28:24.365053 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.365021 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:28:24.365053 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.365052 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:28:24.553573 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.553535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wjxm" event={"ID":"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139","Type":"ContainerStarted","Data":"e199cd9d73299ccd0a292abe4beb7875d5c2a316ad3fcb174ee1513572fd9215"} Nov 03 20:28:24.555953 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.555923 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wwvxj" event={"ID":"9d5cb641-099e-4366-9d79-85d72a0e1a90","Type":"ContainerStarted","Data":"eedcd628dfe6ef5d32211983fc02794d7429f5e55e6b22b186033a965c083054"} Nov 03 20:28:24.558086 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.558059 2578 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="75f7ce5f75a99f9e4164cddafa6f434ba1bb64540c02a297d4d6c117e62a5951" exitCode=0 Nov 03 20:28:24.558204 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.558145 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerDied","Data":"75f7ce5f75a99f9e4164cddafa6f434ba1bb64540c02a297d4d6c117e62a5951"} Nov 03 20:28:24.561963 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.561940 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" event={"ID":"228f5775-1d78-4310-a6d4-fccb3f18a4f4","Type":"ContainerStarted","Data":"ad600945f688ab4811658be9a2451ab76c085a2887bde77c029a4f77b7472212"} Nov 03 20:28:24.659370 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.659318 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4wjxm" podStartSLOduration=3.774502491 podStartE2EDuration="12.659306354s" totalImagesPullingTime="8.884803863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:24.658722442 +0000 UTC m=+12.848119361" watchObservedRunningTime="2026-11-03 20:28:24.659306354 +0000 UTC m=+12.848703256" Nov 03 20:28:24.877084 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:24.877034 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wwvxj" podStartSLOduration=4.229510168 podStartE2EDuration="12.877019427s" totalImagesPullingTime="8.647509259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:24.876633558 +0000 UTC m=+13.066030463" watchObservedRunningTime="2026-11-03 20:28:24.877019427 +0000 UTC m=+13.066416337" Nov 03 20:28:25.304005 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.303980 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:25.304589 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.304572 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:25.439146 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.438684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:25.439146 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.438704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:25.439146 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.438715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:25.439146 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:25.438833 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:25.439146 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:25.438895 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:25.439146 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:25.438983 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:25.565492 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.565460 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gcz42" event={"ID":"e9fa46f4-4f1e-4092-a839-1dd075263b50","Type":"ContainerStarted","Data":"60ed21bd401df586ee4f7aa64b7a885eec99a740871e0178d75c95ef249b08db"} Nov 03 20:28:25.568285 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.568254 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" event={"ID":"228f5775-1d78-4310-a6d4-fccb3f18a4f4","Type":"ContainerStarted","Data":"f940d59f7d6efe71abb2312812ab237f03724e80f6b89f76070d21a591db82b0"} Nov 03 20:28:25.671030 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:25.670974 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gcz42" podStartSLOduration=5.221884548 podStartE2EDuration="13.670959238s" totalImagesPullingTime="8.44907469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:25.670440308 +0000 UTC m=+13.859837213" watchObservedRunningTime="2026-11-03 20:28:25.670959238 +0000 UTC m=+13.860356144" Nov 03 20:28:26.572523 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:26.572479 2578 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="a2355025b14a066252a542c6a0d323e43f362541f4a94a28f1ce38c9323234b5" exitCode=0 Nov 03 20:28:26.573229 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:26.572573 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerDied","Data":"a2355025b14a066252a542c6a0d323e43f362541f4a94a28f1ce38c9323234b5"} Nov 03 20:28:26.574540 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:26.574481 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-92trd" event={"ID":"ba768925-3dbc-4dcb-a35c-54e20f5bf36b","Type":"ContainerStarted","Data":"94bb8acd6b6e90913de6a9a0e1b53025f47188ebfe8fff89c041809a98aabfa8"} Nov 03 20:28:26.749786 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:26.749732 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" podStartSLOduration=4.693323208 podStartE2EDuration="14.749718206s" totalImagesPullingTime="10.056394998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:25.783698716 +0000 UTC m=+13.973095622" watchObservedRunningTime="2026-11-03 20:28:26.749718206 +0000 UTC m=+14.939115110" Nov 03 20:28:27.438528 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:27.438496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:27.438700 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:27.438617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:27.438700 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:27.438631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:27.438700 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:27.438636 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:27.438816 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:27.438705 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:27.438816 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:27.438781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:28.581882 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:28.581658 2578 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="22dd297b5c5eb550a9e859e0d6a16a4a68d35546e1957d9df1c23d86edb9dedf" exitCode=0 Nov 03 20:28:28.582361 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:28.581744 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerDied","Data":"22dd297b5c5eb550a9e859e0d6a16a4a68d35546e1957d9df1c23d86edb9dedf"} Nov 03 20:28:28.786153 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:28.786091 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-92trd" podStartSLOduration=6.14961692 podStartE2EDuration="16.786075037s" totalImagesPullingTime="10.636458117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:26.903617276 +0000 UTC m=+15.093014183" watchObservedRunningTime="2026-11-03 20:28:28.786075037 +0000 UTC m=+16.975471944" Nov 03 20:28:29.336376 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:29.336345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:29.336555 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:29.336393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:29.336555 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.336507 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:29.336660 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.336568 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:45.336551308 +0000 UTC m=+33.525948193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:29.336828 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.336507 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:29.336878 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.336864 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret podName:8eddcaf9-30b4-44e8-bc77-24b08f234497 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:45.336850328 +0000 UTC m=+33.526247211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret") pod "global-pull-secret-syncer-xfqt9" (UID: "8eddcaf9-30b4-44e8-bc77-24b08f234497") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:29.437246 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:29.437215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:29.437409 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.437387 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:29.437468 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.437410 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:29.437468 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.437423 2578 projected.go:196] Error preparing data for projected volume kube-api-access-69j74 for pod openshift-network-diagnostics/network-check-target-7v5wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:29.437553 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.437485 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74 podName:662d94e5-925a-4454-af65-8f767b48f365 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:45.437466238 +0000 UTC m=+33.626863135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-69j74" (UniqueName: "kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74") pod "network-check-target-7v5wf" (UID: "662d94e5-925a-4454-af65-8f767b48f365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:29.438271 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:29.438234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:29.438398 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:29.438289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:29.438398 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:29.438378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:29.438494 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.438391 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:29.438494 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.438486 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:29.438574 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:29.438554 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:30.589649 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.589471 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:28:30.590117 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.590040 2578 generic.go:356] "Generic (PLEG): container finished" podID="0568938d-c9e2-4e37-b768-cdeddaacf33e" containerID="888636e7250d02747d4bac990725ab910abdcb32e778d575cd66c10dee803b84" exitCode=1 Nov 03 20:28:30.590117 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.590076 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"0ee42eaea8e9c1991ac754b8f5d3568ac5932228594639d5e1480f57628c9279"} Nov 03 20:28:30.590220 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.590151 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"82dadaf51b487c5402db443feb6523d52e20f85b3bd7911324b85c578674e7d0"} Nov 03 20:28:30.590273 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.590225 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"f5adc381c52f27b47d9e363c991e639729973a5508eb038b8f912d91052a4533"} Nov 03 20:28:30.590273 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.590258 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"2a1171b6396e1f4ad7b24ffccd9e3e2301811270dd617e484f5b9d68cbbfd339"} Nov 03 20:28:30.590345 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.590287 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerDied","Data":"888636e7250d02747d4bac990725ab910abdcb32e778d575cd66c10dee803b84"} Nov 03 20:28:30.590345 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:30.590319 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"aa1a3d0fd1063959319f6a3e9f5a6a15b5d3b12f72b3f0f54c2113fe350c8320"} Nov 03 20:28:31.439109 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:31.439075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:31.439109 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:31.439106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:31.439295 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:31.439237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:31.439295 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:31.439244 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:31.439352 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:31.439304 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:31.439381 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:31.439358 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:31.594689 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:31.594660 2578 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="80556bba002a969a92a0ab0b446101f9e574da426fda410e3fb2551e462adb91" exitCode=0 Nov 03 20:28:31.595141 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:31.594704 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerDied","Data":"80556bba002a969a92a0ab0b446101f9e574da426fda410e3fb2551e462adb91"} Nov 03 20:28:32.599334 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:32.599261 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:28:32.599771 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:32.599588 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"ecbd6529861fe92a1c56e867703e7673efaceaf6b3592b1553f89aa029424014"} Nov 03 20:28:32.602533 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:32.602511 2578 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="a0f20dbaac3831695a857409114692d77ba856015b38986ccf74fbb32c3518c5" exitCode=0 Nov 03 20:28:32.602650 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:32.602545 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerDied","Data":"a0f20dbaac3831695a857409114692d77ba856015b38986ccf74fbb32c3518c5"} Nov 03 20:28:33.439074 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:33.439040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:33.439249 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:33.439104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:33.439249 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:33.439128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:33.439249 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:33.439232 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:33.439361 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:33.439265 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:33.439361 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:33.439340 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:33.607798 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:33.607768 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr82s" event={"ID":"13dda6c1-f004-46b6-928c-47d6de1d58f9","Type":"ContainerStarted","Data":"3ffb5c253d85c098f0659696525b95d5b67bacbd673e7511e59004857b9df58e"} Nov 03 20:28:33.719417 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:33.719343 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jr82s" podStartSLOduration=8.672393257 podStartE2EDuration="21.719331147s" totalImagesPullingTime="13.04693789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:33.718853785 +0000 UTC m=+21.908250689" watchObservedRunningTime="2026-11-03 20:28:33.719331147 +0000 UTC m=+21.908728051" Nov 03 20:28:34.613262 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.613017 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:28:34.613743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.613615 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"3d4f8c8dddff0b5659845367ff3da76e77245f8f40df50a1f83cfb8de20b1911"} Nov 03 20:28:34.614038 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.614016 2578 scope.go:122] "RemoveContainer" containerID="888636e7250d02747d4bac990725ab910abdcb32e778d575cd66c10dee803b84" Nov 03 20:28:34.614193 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.614015 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:34.614193 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.614154 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:34.614314 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.614223 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:34.630365 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.630344 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:34.630454 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:34.630436 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:35.439000 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:35.438972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:35.439345 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:35.439068 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:35.439345 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:35.439075 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:35.439345 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:35.439128 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:35.439345 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:35.439188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:35.439345 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:35.439281 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:35.619773 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:35.619748 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:28:35.620199 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:35.620122 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" event={"ID":"0568938d-c9e2-4e37-b768-cdeddaacf33e","Type":"ContainerStarted","Data":"f11ed175124ee39f67b318bae3d587355909eb3c3eca8a4721421ff58e575beb"} Nov 03 20:28:35.734741 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:35.734663 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" podStartSLOduration=8.000429343 podStartE2EDuration="22.734650568s" totalImagesPullingTime="14.734221225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:35.733949763 +0000 UTC m=+23.923346668" watchObservedRunningTime="2026-11-03 20:28:35.734650568 +0000 UTC m=+23.924047500" Nov 03 20:28:37.018928 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:37.018898 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfqt9"] Nov 03 20:28:37.019431 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:37.019010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:37.019431 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:37.019126 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:37.021994 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:37.021967 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7v5wf"] Nov 03 20:28:37.022097 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:37.022058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:37.022187 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:37.022146 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:37.065504 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:37.065480 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xqvnn"] Nov 03 20:28:37.065595 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:37.065557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:37.065650 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:37.065633 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:38.441873 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:38.441848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:38.442188 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:38.441972 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:38.442188 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:38.442000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:38.442188 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:38.442048 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:39.438373 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:39.438349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:39.438511 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:39.438440 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:40.441661 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:40.441639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:40.441967 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:40.441740 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:40.441967 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:40.441789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:40.441967 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:40.441880 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:41.372648 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:41.372621 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:41.372819 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:41.372728 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:28:41.373227 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:41.373208 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:41.438555 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:41.438538 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:41.438637 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:41.438613 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v5wf" podUID="662d94e5-925a-4454-af65-8f767b48f365" Nov 03 20:28:42.439922 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.439759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:42.440354 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.439851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:42.440354 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:42.439990 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfqt9" podUID="8eddcaf9-30b4-44e8-bc77-24b08f234497" Nov 03 20:28:42.440354 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:42.440151 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqvnn" podUID="e55a9e03-4858-42b2-a27a-a9533e404588" Nov 03 20:28:42.606402 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.606340 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeReady" Nov 03 20:28:42.606531 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.606427 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:28:42.751378 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.751351 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lrdnf"] Nov 03 20:28:42.776248 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.776222 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wcndc"] Nov 03 20:28:42.776386 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.776374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:42.790069 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.790049 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9k29p"] Nov 03 20:28:42.790187 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.790149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:42.803911 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.803893 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lrdnf"] Nov 03 20:28:42.804013 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.803928 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wcndc"] Nov 03 20:28:42.804013 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.803960 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9k29p"] Nov 03 20:28:42.804013 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.803959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:42.825948 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.825932 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:28:42.826042 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.825953 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:28:42.826042 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.825951 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:28:42.826042 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.826015 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fpmzx\"" Nov 03 20:28:42.837100 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.837052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40a6b42d-1468-4f9b-85f3-4243a25e01ac-cert\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:42.837182 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.837135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9f5\" (UniqueName: \"kubernetes.io/projected/40a6b42d-1468-4f9b-85f3-4243a25e01ac-kube-api-access-wr9f5\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:42.839134 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.839118 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:28:42.839227 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.839147 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:28:42.839227 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.839212 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:28:42.839227 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.839212 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:28:42.839337 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.839242 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntq9n\"" Nov 03 20:28:42.853992 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.853978 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pqh8h\"" Nov 03 20:28:42.927068 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.927047 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:28:42.927155 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.927080 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:28:42.938204 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:42.938289 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd648a5-3a08-42c1-b8a3-e14ae414263c-crio-socket\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:42.938289 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2964a2b0-45fb-4282-8cde-f2ef72de59c5-tmp-dir\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:42.938366 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:42.938401 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8gd\" (UniqueName: \"kubernetes.io/projected/2964a2b0-45fb-4282-8cde-f2ef72de59c5-kube-api-access-rm8gd\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:42.938441 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40a6b42d-1468-4f9b-85f3-4243a25e01ac-cert\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:42.938482 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9f5\" (UniqueName: \"kubernetes.io/projected/40a6b42d-1468-4f9b-85f3-4243a25e01ac-kube-api-access-wr9f5\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:42.938482 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:42.938563 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bgw\" (UniqueName: \"kubernetes.io/projected/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-api-access-x9bgw\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:42.938648 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2964a2b0-45fb-4282-8cde-f2ef72de59c5-config-volume\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:42.938692 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.938666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd648a5-3a08-42c1-b8a3-e14ae414263c-data-volume\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:42.942123 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.942103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40a6b42d-1468-4f9b-85f3-4243a25e01ac-cert\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:42.993189 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:42.993150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9f5\" (UniqueName: \"kubernetes.io/projected/40a6b42d-1468-4f9b-85f3-4243a25e01ac-kube-api-access-wr9f5\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:43.039781 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.039762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd648a5-3a08-42c1-b8a3-e14ae414263c-crio-socket\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.039869 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.039786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2964a2b0-45fb-4282-8cde-f2ef72de59c5-tmp-dir\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.039869 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.039805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.039869 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.039822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8gd\" (UniqueName: \"kubernetes.io/projected/2964a2b0-45fb-4282-8cde-f2ef72de59c5-kube-api-access-rm8gd\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.040021 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.039908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.040021 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.039938 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:43.040021 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.039946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bgw\" (UniqueName: \"kubernetes.io/projected/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-api-access-x9bgw\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.040021 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.039991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd648a5-3a08-42c1-b8a3-e14ae414263c-crio-socket\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.040021 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.040001 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:43.539983153 +0000 UTC m=+31.729380054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:43.040021 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.040017 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:43.040326 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.040034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2964a2b0-45fb-4282-8cde-f2ef72de59c5-config-volume\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.040326 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.040065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd648a5-3a08-42c1-b8a3-e14ae414263c-data-volume\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.040326 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.040079 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:43.540062345 +0000 UTC m=+31.729459242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : configmap "kube-rbac-proxy" not found Nov 03 20:28:43.040326 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.040078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2964a2b0-45fb-4282-8cde-f2ef72de59c5-tmp-dir\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.040326 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.040172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.040326 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.040234 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:43.040326 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.040288 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:43.540272253 +0000 UTC m=+31.729669152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:43.040532 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.040349 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd648a5-3a08-42c1-b8a3-e14ae414263c-data-volume\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.040582 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.040567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2964a2b0-45fb-4282-8cde-f2ef72de59c5-config-volume\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.093993 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.093968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bgw\" (UniqueName: \"kubernetes.io/projected/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-api-access-x9bgw\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.094115 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.094100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8gd\" (UniqueName: \"kubernetes.io/projected/2964a2b0-45fb-4282-8cde-f2ef72de59c5-kube-api-access-rm8gd\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.098392 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.098368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:43.412341 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.412308 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n"] Nov 03 20:28:43.424543 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.424518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" Nov 03 20:28:43.438568 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.438547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:43.468850 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.468828 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n"] Nov 03 20:28:43.473647 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.473627 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:43.473647 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.473630 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:28:43.473754 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.473692 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8hb94\"" Nov 03 20:28:43.549187 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.549120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcf4v\" (UniqueName: \"kubernetes.io/projected/c020730b-79dc-43cb-9732-2102974c80c0-kube-api-access-vcf4v\") pod \"migrator-66df5d496b-59x5n\" (UID: \"c020730b-79dc-43cb-9732-2102974c80c0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" Nov 03 20:28:43.549187 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.549154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:43.549319 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.549238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.549319 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.549273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:43.549398 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.549330 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:43.549398 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.549342 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:43.549398 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.549350 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:43.549398 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.549383 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.549369054 +0000 UTC m=+32.738765941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:43.549398 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.549396 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.549390238 +0000 UTC m=+32.738787121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:43.549561 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:43.549407 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.549402299 +0000 UTC m=+32.738799182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : configmap "kube-rbac-proxy" not found Nov 03 20:28:43.551656 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.551638 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lrdnf"] Nov 03 20:28:43.560749 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:43.560728 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a6b42d_1468_4f9b_85f3_4243a25e01ac.slice/crio-680a8b70eddda0b83d0621e15bf125399f531be96ce3e724bc3f79b7df1df697 WatchSource:0}: Error finding container 680a8b70eddda0b83d0621e15bf125399f531be96ce3e724bc3f79b7df1df697: Status 404 returned error can't find the container with id 680a8b70eddda0b83d0621e15bf125399f531be96ce3e724bc3f79b7df1df697 Nov 03 20:28:43.574131 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.574113 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:28:43.574263 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.574251 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jxjwz\"" Nov 03 20:28:43.575615 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.575603 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:28:43.637977 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.637956 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lrdnf" event={"ID":"40a6b42d-1468-4f9b-85f3-4243a25e01ac","Type":"ContainerStarted","Data":"680a8b70eddda0b83d0621e15bf125399f531be96ce3e724bc3f79b7df1df697"} Nov 03 20:28:43.650359 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.650339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcf4v\" (UniqueName: \"kubernetes.io/projected/c020730b-79dc-43cb-9732-2102974c80c0-kube-api-access-vcf4v\") pod \"migrator-66df5d496b-59x5n\" (UID: \"c020730b-79dc-43cb-9732-2102974c80c0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" Nov 03 20:28:43.703847 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.703823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcf4v\" (UniqueName: \"kubernetes.io/projected/c020730b-79dc-43cb-9732-2102974c80c0-kube-api-access-vcf4v\") pod \"migrator-66df5d496b-59x5n\" (UID: \"c020730b-79dc-43cb-9732-2102974c80c0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" Nov 03 20:28:43.743595 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:43.743574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" Nov 03 20:28:44.121017 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.120954 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n"] Nov 03 20:28:44.128564 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:44.128537 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc020730b_79dc_43cb_9732_2102974c80c0.slice/crio-61d8f3d671acb9b9c0340abfe614b18c7cdad38c243dc3461348d27fe6fc2d59 WatchSource:0}: Error finding container 61d8f3d671acb9b9c0340abfe614b18c7cdad38c243dc3461348d27fe6fc2d59: Status 404 returned error can't find the container with id 61d8f3d671acb9b9c0340abfe614b18c7cdad38c243dc3461348d27fe6fc2d59 Nov 03 20:28:44.441070 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.441051 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:44.441256 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.441240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:44.492281 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.492261 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:28:44.492601 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.492343 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h9ssl\"" Nov 03 20:28:44.492601 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.492375 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kpkvl\"" Nov 03 20:28:44.492601 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.492420 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:28:44.556743 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.556719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:44.556826 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.556757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:44.556826 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.556779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:44.556914 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:44.556855 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:44.556914 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:44.556885 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:44.556914 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:44.556902 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.556886519 +0000 UTC m=+34.746283417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : configmap "kube-rbac-proxy" not found Nov 03 20:28:44.557017 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:44.556929 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.556917388 +0000 UTC m=+34.746314271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:44.557017 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:44.556939 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:44.557017 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:44.556980 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.556967763 +0000 UTC m=+34.746364660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:44.641427 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:44.641293 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" event={"ID":"c020730b-79dc-43cb-9732-2102974c80c0","Type":"ContainerStarted","Data":"61d8f3d671acb9b9c0340abfe614b18c7cdad38c243dc3461348d27fe6fc2d59"} Nov 03 20:28:45.362503 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.362473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:45.362647 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.362565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:45.362689 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:45.362660 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:45.362741 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:45.362729 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:17.362710766 +0000 UTC m=+65.552107663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : secret "metrics-daemon-secret" not found Nov 03 20:28:45.364914 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.364896 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:45.380978 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.380956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:45.463670 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.463647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:45.466456 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.466431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:45.554828 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.554807 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:45.752848 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.752820 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfqt9"] Nov 03 20:28:45.760372 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:45.760312 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eddcaf9_30b4_44e8_bc77_24b08f234497.slice/crio-51b0b83934216acaba00407086965a33988662f7d94c03bd74888c594ada8a13 WatchSource:0}: Error finding container 51b0b83934216acaba00407086965a33988662f7d94c03bd74888c594ada8a13: Status 404 returned error can't find the container with id 51b0b83934216acaba00407086965a33988662f7d94c03bd74888c594ada8a13 Nov 03 20:28:45.947465 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:45.947432 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7v5wf"] Nov 03 20:28:46.042446 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.042417 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8mp2t"] Nov 03 20:28:46.073880 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.073860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.098248 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.098225 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8mp2t"] Nov 03 20:28:46.124038 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.124011 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:28:46.124150 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.124105 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:28:46.144373 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.144349 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:28:46.144559 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.144539 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:28:46.144623 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.144534 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2t6xn\"" Nov 03 20:28:46.169071 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.169050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-cabundle\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.169198 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.169139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrcj\" (UniqueName: \"kubernetes.io/projected/44a82b10-e0ed-4945-8f9a-845487bbb556-kube-api-access-vnrcj\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.169255 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.169226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-key\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.270471 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.270393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-cabundle\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.270471 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.270445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrcj\" (UniqueName: \"kubernetes.io/projected/44a82b10-e0ed-4945-8f9a-845487bbb556-kube-api-access-vnrcj\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.270761 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.270504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-key\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.271255 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.271223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-cabundle\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.273677 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.273652 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-key\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.324048 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.324025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrcj\" (UniqueName: \"kubernetes.io/projected/44a82b10-e0ed-4945-8f9a-845487bbb556-kube-api-access-vnrcj\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.397005 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.396972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:46.572421 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.572325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:46.572421 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.572398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:46.572944 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.572438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:46.572944 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:46.572522 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:28:46.572944 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:46.572585 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.572567739 +0000 UTC m=+38.761964628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : configmap "kube-rbac-proxy" not found Nov 03 20:28:46.572944 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:46.572590 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:46.572944 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:46.572649 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.572635589 +0000 UTC m=+38.762032490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:46.572944 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:46.572521 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:46.572944 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:46.572685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.572675723 +0000 UTC m=+38.762072623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:46.647885 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.647857 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7v5wf" event={"ID":"662d94e5-925a-4454-af65-8f767b48f365","Type":"ContainerStarted","Data":"d3ce7db4b3804e84f9642e66b86195b7a91f1c9ac2e3dd86e63c0085bc12f480"} Nov 03 20:28:46.648898 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.648874 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfqt9" event={"ID":"8eddcaf9-30b4-44e8-bc77-24b08f234497","Type":"ContainerStarted","Data":"51b0b83934216acaba00407086965a33988662f7d94c03bd74888c594ada8a13"} Nov 03 20:28:46.781281 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:46.781253 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8mp2t"] Nov 03 20:28:46.907096 ip-10-0-134-71 kubenswrapper[2578]: W1103 20:28:46.907068 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a82b10_e0ed_4945_8f9a_845487bbb556.slice/crio-a2828a6e0d67d847662103ad66d1429042509d4e6ca0ec9e06ed41c1bde1c1e4 WatchSource:0}: Error finding container a2828a6e0d67d847662103ad66d1429042509d4e6ca0ec9e06ed41c1bde1c1e4: Status 404 returned error can't find the container with id a2828a6e0d67d847662103ad66d1429042509d4e6ca0ec9e06ed41c1bde1c1e4 Nov 03 20:28:47.655192 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:47.654989 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" event={"ID":"c020730b-79dc-43cb-9732-2102974c80c0","Type":"ContainerStarted","Data":"6f2bb002e2c3cad18d11bbdb95611b7ce893db14a9af6854854bd36b6501c073"} Nov 03 20:28:47.657275 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:47.657228 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" event={"ID":"44a82b10-e0ed-4945-8f9a-845487bbb556","Type":"ContainerStarted","Data":"a2828a6e0d67d847662103ad66d1429042509d4e6ca0ec9e06ed41c1bde1c1e4"} Nov 03 20:28:47.809040 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:47.808980 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" podStartSLOduration=2.009779429 podStartE2EDuration="4.808962281s" totalImagesPullingTime="2.799182852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:47.761744546 +0000 UTC m=+35.951141452" watchObservedRunningTime="2026-11-03 20:28:47.808962281 +0000 UTC m=+35.998359188" Nov 03 20:28:48.663463 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:48.663266 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lrdnf" event={"ID":"40a6b42d-1468-4f9b-85f3-4243a25e01ac","Type":"ContainerStarted","Data":"73286badedbf555d6abce7db826b5b4140e7d45b902d930716bac89737218828"} Nov 03 20:28:48.770664 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:48.770600 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lrdnf" podStartSLOduration=1.9319401109999998 podStartE2EDuration="6.770581234s" totalImagesPullingTime="4.838641123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:48.769654306 +0000 UTC m=+36.959051211" watchObservedRunningTime="2026-11-03 20:28:48.770581234 +0000 UTC m=+36.959978138" Nov 03 20:28:50.607906 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:50.607870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:50.607906 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:50.607923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:50.608400 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:50.607957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:50.608577 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:50.608546 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:50.608703 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:50.608622 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:58.60860187 +0000 UTC m=+46.797998757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:50.608703 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:50.608552 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:50.608812 ip-10-0-134-71 kubenswrapper[2578]: E1103 20:28:50.608708 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:58.608690257 +0000 UTC m=+46.798087175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:50.608812 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:50.608766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:50.822821 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:50.820307 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wjxm_aa839ba6-7b29-41c8-bd50-a7cd5b1ac139/dns-node-resolver/0.log" Nov 03 20:28:51.672507 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:51.672473 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7v5wf" event={"ID":"662d94e5-925a-4454-af65-8f767b48f365","Type":"ContainerStarted","Data":"1f9e00ece43728d9f8dbff18210c87974e8b2cbe998c0e5c8d54730f7b4b7f2d"} Nov 03 20:28:51.673067 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:51.672625 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:51.780504 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:51.780452 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7v5wf" podStartSLOduration=34.994186985 podStartE2EDuration="39.780436241s" totalImagesPullingTime="4.786249256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:51.780001242 +0000 UTC m=+39.969398147" watchObservedRunningTime="2026-11-03 20:28:51.780436241 +0000 UTC m=+39.969833147" Nov 03 20:28:52.016074 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:52.016010 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dg8z2_11a94da7-4353-4baf-9cd9-e585ff91ac82/node-ca/0.log" Nov 03 20:28:52.676532 ip-10-0-134-71 kubenswrapper[2578]: I1103 20:28:52.676490 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfqt9" event={"ID":"8eddcaf9-30b4-44e8-bc77-24b08f234497","Type":"ContainerStarted","Data":"51fb099d1287e74ae09006edd5be42249fa4c02f51183a265a168201ac9674d3"} Nov 03 20:28:52.687782 ip-10-0-134-71 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:28:52.697758 ip-10-0-134-71 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:28:52.698003 ip-10-0-134-71 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:28:52.698351 ip-10-0-134-71 systemd[1]: kubelet.service: Consumed 2.264s CPU time. Nov 03 20:28:52.717633 ip-10-0-134-71 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:28:52.813834 ip-10-0-134-71 kubenswrapper[4227]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:52.813834 ip-10-0-134-71 kubenswrapper[4227]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:28:52.813834 ip-10-0-134-71 kubenswrapper[4227]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:52.813834 ip-10-0-134-71 kubenswrapper[4227]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:52.817075 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.816999 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.817075 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817072 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.817075 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817076 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.817075 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817079 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817083 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817086 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817089 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817091 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817094 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817096 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817099 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817102 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817104 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817107 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817110 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817113 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817115 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817118 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817120 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817123 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817126 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817129 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.817272 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817132 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817134 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817137 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817139 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817142 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817145 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817147 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817150 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817153 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817171 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817174 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817178 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817182 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817185 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817188 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817191 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817195 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817199 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817203 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.817830 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817206 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817209 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817212 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817214 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817217 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817219 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817222 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817225 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817227 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817230 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817234 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817237 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817240 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817243 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817245 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817248 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817250 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817253 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817255 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.818328 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817258 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817260 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817263 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817266 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817268 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817271 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817274 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817277 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817279 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817282 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817285 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817288 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817291 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817293 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817296 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817298 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817300 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817303 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817305 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817308 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.818805 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817310 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817313 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817315 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817319 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817323 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817325 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817328 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817330 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817333 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817335 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817338 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817340 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817343 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817345 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817349 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817352 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817354 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817357 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817359 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817362 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.819314 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817365 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817367 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817370 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817372 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817375 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.817377 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818143 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818150 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818154 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818174 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818178 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818181 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818183 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818186 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818189 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818194 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818199 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818203 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818206 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.819812 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818209 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818212 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818215 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818217 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818220 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818224 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818228 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818231 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818234 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818237 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818239 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818242 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818245 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818247 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818250 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818253 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818255 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818258 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818261 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.820303 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818263 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818266 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818269 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818271 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818274 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818277 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818279 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818282 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818284 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818287 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818290 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818293 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818296 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818298 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818301 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818303 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818306 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818309 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818312 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818314 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.820783 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818317 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818320 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818323 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818325 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818328 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818331 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818333 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818337 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818339 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818342 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818345 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818348 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818350 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818353 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818355 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818358 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818361 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818363 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818366 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818368 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.821294 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818371 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818373 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818376 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818379 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818382 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818385 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818387 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818390 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818392 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818395 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818397 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818400 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818403 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818405 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818408 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818412 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818414 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818417 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818420 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.821779 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818422 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818425 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818428 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818430 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818433 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818435 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818438 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818440 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818443 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818445 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818448 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818450 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818453 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818455 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818458 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818938 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818943 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818946 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818949 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818951 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.822263 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818954 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818957 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818959 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818962 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818965 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818970 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818973 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818976 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818979 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818981 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818986 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818988 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818991 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818993 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818996 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.818999 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819001 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819004 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819007 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819009 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.822791 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819012 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819014 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819017 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819029 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819032 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819035 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819038 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819040 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819043 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819045 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819049 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819051 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819054 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819056 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819059 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819062 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819064 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819068 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819071 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.823305 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819073 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819076 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819085 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819091 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819094 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819097 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819099 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819102 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819105 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819107 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819110 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819112 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819115 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819117 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819120 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819122 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819125 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819127 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819130 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.823761 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819133 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819135 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819138 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819140 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819143 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819146 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819149 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819151 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819154 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819169 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819173 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819179 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819182 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819185 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819188 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819190 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819194 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819197 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819200 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.824505 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819202 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819205 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819207 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819210 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819213 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819216 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819218 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819221 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819224 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819227 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819230 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819232 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819235 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819237 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819240 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819242 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819245 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819247 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819250 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819253 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.825334 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819255 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819258 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819261 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.819264 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819375 4227 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819385 4227 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819394 4227 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819401 4227 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819408 4227 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819414 4227 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819418 4227 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819422 4227 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819425 4227 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819429 4227 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819432 4227 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819436 4227 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819439 4227 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819442 4227 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819445 4227 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819448 4227 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819451 4227 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819454 4227 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819458 4227 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:28:52.826218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819461 4227 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819464 4227 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819467 4227 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819470 4227 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819474 4227 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819477 4227 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819480 4227 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819484 4227 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819487 4227 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819491 4227 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819494 4227 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819497 4227 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819500 4227 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819505 4227 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819508 4227 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819512 4227 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819515 4227 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819518 4227 flags.go:64] FLAG: --enable-server="true" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819523 4227 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819527 4227 flags.go:64] FLAG: --event-burst="100" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819531 4227 flags.go:64] FLAG: --event-qps="50" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819534 4227 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819537 4227 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819540 4227 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:28:52.827239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819544 4227 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819547 4227 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819550 4227 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819554 4227 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819557 4227 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819560 4227 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819563 4227 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819566 4227 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819569 4227 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819572 4227 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819575 4227 flags.go:64] FLAG: --feature-gates="" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819579 4227 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819582 4227 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819585 4227 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819588 4227 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819591 4227 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819594 4227 flags.go:64] FLAG: --help="false" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819597 4227 flags.go:64] FLAG: --hostname-override="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819601 4227 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819604 4227 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819607 4227 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819611 4227 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819615 4227 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:28:52.828351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819618 4227 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819621 4227 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819624 4227 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819629 4227 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819632 4227 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819635 4227 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819638 4227 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819641 4227 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819644 4227 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819647 4227 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819650 4227 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819653 4227 flags.go:64] FLAG: --lock-file="" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819655 4227 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819658 4227 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819661 4227 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819667 4227 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819670 4227 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819673 4227 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819676 4227 flags.go:64] FLAG: --logging-format="text" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819679 4227 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819682 4227 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819685 4227 flags.go:64] FLAG: --manifest-url="" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819688 4227 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819692 4227 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819695 4227 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:28:52.829561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819699 4227 flags.go:64] FLAG: --max-pods="110" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819702 4227 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819705 4227 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819709 4227 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819712 4227 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819716 4227 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819719 4227 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819722 4227 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819729 4227 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819732 4227 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819736 4227 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819739 4227 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819742 4227 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819745 4227 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819748 4227 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819751 4227 flags.go:64] FLAG: --port="10250" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819754 4227 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819757 4227 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a97411e7ded09ea5" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819760 4227 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819763 4227 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819766 4227 flags.go:64] FLAG: --register-node="true" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819769 4227 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819773 4227 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819776 4227 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819779 4227 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819781 4227 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:28:52.830669 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819785 4227 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819788 4227 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819791 4227 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819794 4227 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819797 4227 flags.go:64] FLAG: --runonce="false" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819800 4227 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819803 4227 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819806 4227 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819809 4227 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819812 4227 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819815 4227 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819820 4227 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819824 4227 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819827 4227 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819830 4227 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819832 4227 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819836 4227 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819840 4227 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819843 4227 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819845 4227 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819851 4227 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819854 4227 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819857 4227 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819861 4227 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819864 4227 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:28:52.831840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819867 4227 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819869 4227 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819873 4227 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819876 4227 flags.go:64] FLAG: --v="2" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819880 4227 flags.go:64] FLAG: --version="false" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819884 4227 flags.go:64] FLAG: --vmodule="" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819888 4227 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.819891 4227 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820025 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820029 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820032 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820035 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820037 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820040 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820043 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820045 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820048 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820051 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820057 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820060 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820063 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.832945 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820066 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820069 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820072 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820075 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820078 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820080 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820083 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820086 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820088 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820091 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820094 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820097 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820099 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820102 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820105 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820107 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820110 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820112 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820115 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.833756 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820117 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820120 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820123 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820125 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820128 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820130 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820133 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820135 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820138 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820141 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820145 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820149 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820151 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820154 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820168 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820173 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820175 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820178 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820181 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820183 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.834419 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820186 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820189 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820192 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820194 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820197 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820200 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820202 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820205 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820207 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820210 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820213 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820216 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820218 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820221 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820223 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820226 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820228 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820231 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820234 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820236 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.835057 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820239 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820241 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820246 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820248 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820251 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820254 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820257 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820260 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820263 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820265 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820269 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820273 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820276 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820278 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820281 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820284 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820286 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820289 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820291 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820294 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.835973 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820296 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820299 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820302 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820304 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820307 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820310 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820313 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820315 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820318 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820322 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820326 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820329 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820332 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820334 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.820360 4227 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:52.836571 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820414 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820419 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820422 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820425 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820429 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820432 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820435 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820437 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820440 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820442 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820445 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820447 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820450 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820452 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820455 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820457 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820460 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820462 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820466 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.836991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820470 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820473 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820476 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820479 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820481 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820484 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820486 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820489 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820491 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820494 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820496 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820499 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820501 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820504 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820506 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820509 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820512 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820516 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820518 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820521 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.837734 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820523 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820526 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820528 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820531 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820533 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820536 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820538 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820541 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820543 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820546 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820548 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820551 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820553 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820556 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820558 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820561 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820564 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820566 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820569 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.838550 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820572 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820574 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820577 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820579 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820581 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820584 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820587 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820589 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820592 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820594 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820600 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820602 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820605 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820607 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820610 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820612 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820614 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820617 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820619 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820622 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.839333 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820624 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820627 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820630 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820632 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820635 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820637 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820639 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820642 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820645 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820647 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820650 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820652 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820655 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820657 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820660 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820662 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820665 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820667 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820670 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820672 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.840186 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820675 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820677 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820681 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820684 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820686 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820689 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820691 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.820695 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.826346 4227 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.826364 4227 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.826388 4227 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.826396 4227 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826478 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826486 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826492 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826496 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826501 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826506 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826510 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.841036 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826517 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826523 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826529 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826533 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826538 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826542 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826547 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826551 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826556 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826560 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826564 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826569 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826574 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826578 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826582 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826587 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826591 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826596 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826601 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826605 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.841867 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826609 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826616 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826623 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826628 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826633 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826638 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826642 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826647 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826651 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826655 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826660 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826664 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826669 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826673 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826678 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826683 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826687 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826691 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.842811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826695 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826700 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826704 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826709 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826713 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826717 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826722 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826726 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826731 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826735 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826739 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826743 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826748 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826752 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826756 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826760 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826764 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826769 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826774 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826779 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.843525 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826783 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826787 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826791 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826796 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826800 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826804 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826809 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826814 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826818 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826823 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826827 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826831 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826836 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826840 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826844 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826849 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826853 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826857 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826862 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.844026 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826866 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826871 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826875 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826880 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826884 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826888 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826892 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826897 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826901 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826905 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826910 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826915 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826919 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826923 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826927 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826932 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826936 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826941 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826945 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826950 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.844520 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826954 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.826958 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.826998 4227 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827223 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827232 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827238 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827243 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827248 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827252 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827256 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827260 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827265 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827269 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827274 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827278 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827282 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.845000 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827286 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827291 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827295 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827299 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827303 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827307 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827312 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827316 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827321 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827326 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827330 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827335 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827339 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827343 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827347 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827351 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827356 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827360 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827364 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827368 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.845429 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827373 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827377 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827381 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827386 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827390 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827394 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827398 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827403 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827407 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827411 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827416 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827420 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827425 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827429 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827433 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827437 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827441 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827446 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827450 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827454 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.845913 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827459 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827464 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827469 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827473 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827477 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827481 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827485 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827489 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827494 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827498 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827503 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827510 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827514 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827519 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827523 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827528 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827535 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827541 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827546 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.846448 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827550 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827555 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827559 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827563 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827568 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827572 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827576 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827581 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827585 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827590 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827594 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827599 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827603 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827607 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827611 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827616 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827620 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827624 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827628 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827632 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.846899 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827637 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827641 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827645 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827649 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827656 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827661 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827667 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827672 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827677 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827681 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827686 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827690 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827695 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827699 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.827738 4227 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:52.847427 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827794 4227 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827801 4227 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827806 4227 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827811 4227 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827816 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827821 4227 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827825 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827829 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827833 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827837 4227 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827842 4227 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827846 4227 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827850 4227 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827854 4227 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827859 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827864 4227 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827868 4227 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827872 4227 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827876 4227 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:52.847811 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827881 4227 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827885 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827889 4227 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827895 4227 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827899 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827903 4227 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827908 4227 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827912 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827916 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827920 4227 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827924 4227 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827929 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827933 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827937 4227 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827941 4227 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827945 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827950 4227 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827954 4227 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827958 4227 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:52.848293 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827963 4227 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827967 4227 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827971 4227 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827975 4227 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827980 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827983 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827993 4227 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.827998 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828003 4227 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828008 4227 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828012 4227 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828016 4227 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828020 4227 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828025 4227 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828029 4227 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828034 4227 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828040 4227 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828045 4227 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828049 4227 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828053 4227 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:52.849051 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828057 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828062 4227 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828066 4227 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828070 4227 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828074 4227 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828082 4227 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828088 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828093 4227 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828098 4227 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828102 4227 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828108 4227 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828112 4227 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828116 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828121 4227 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828125 4227 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828130 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828134 4227 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828138 4227 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828144 4227 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:52.849912 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828149 4227 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828153 4227 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828172 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828177 4227 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828181 4227 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828185 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828189 4227 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828194 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828198 4227 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828204 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828208 4227 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828212 4227 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828217 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828221 4227 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828225 4227 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828231 4227 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828237 4227 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828242 4227 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828246 4227 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:52.850738 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828251 4227 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828257 4227 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828261 4227 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828266 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828270 4227 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828276 4227 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828281 4227 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828287 4227 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828291 4227 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:28:52.828295 4227 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.828364 4227 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.829844 4227 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.829916 4227 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.830186 4227 server.go:1014] "Starting client certificate rotation" Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.830318 4227 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.830369 4227 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:12 +0000 UTC" deadline="2028-07-03 12:34:08.84139027 +0000 UTC" Nov 03 20:28:52.851633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.830404 4227 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14584h5m16.010987882s" Nov 03 20:28:52.852320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.830861 4227 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:52.852320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.832460 4227 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:52.852320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.834460 4227 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:28:52.852320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.836312 4227 log.go:25] "Validated CRI v1 image API" Nov 03 20:28:52.852320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.837105 4227 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:28:52.852320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.840184 4227 fs.go:135] Filesystem UUIDs: map[026769af-2496-4d2e-b354-bfb2edc1d8a8:/dev/nvme0n1p3 2b2391b5-24a0-402f-95c1-d0456070e434:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:28:52.852604 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.840212 4227 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a12e8b1edba5656cb5309eb74313ebe671e47c7871aa043e366a985ca8bf157/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a12e8b1edba5656cb5309eb74313ebe671e47c7871aa043e366a985ca8bf157/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a369a7f73a85eef1737426f905fc9b1695ba79250be6e1db369d4c08e80d141/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a369a7f73a85eef1737426f905fc9b1695ba79250be6e1db369d4c08e80d141/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ed7e4eb9858d00756e506026c660a1d367c159e38c375d1a78fe709a4ba8a68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ed7e4eb9858d00756e506026c660a1d367c159e38c375d1a78fe709a4ba8a68/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12d6fa1c8cce7fb251045fd4e88e38b30b084792c3c99018abbbec3602f484c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12d6fa1c8cce7fb251045fd4e88e38b30b084792c3c99018abbbec3602f484c0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41f236f00d4432250b764f7c2e71d99483e2036dab2684cbb0a9ecc4de46c278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41f236f00d4432250b764f7c2e71d99483e2036dab2684cbb0a9ecc4de46c278/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51b0b83934216acaba00407086965a33988662f7d94c03bd74888c594ada8a13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51b0b83934216acaba00407086965a33988662f7d94c03bd74888c594ada8a13/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f0b001be9f8130d7f4d6fd8eea51504a326ee41258e2dd23539a960934cd85e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f0b001be9f8130d7f4d6fd8eea51504a326ee41258e2dd23539a960934cd85e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61d8f3d671acb9b9c0340abfe614b18c7cdad38c243dc3461348d27fe6fc2d59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61d8f3d671acb9b9c0340abfe614b18c7cdad38c243dc3461348d27fe6fc2d59/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/680a8b70eddda0b83d0621e15bf125399f531be96ce3e724bc3f79b7df1df697/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/680a8b70eddda0b83d0621e15bf125399f531be96ce3e724bc3f79b7df1df697/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2828a6e0d67d847662103ad66d1429042509d4e6ca0ec9e06ed41c1bde1c1e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2828a6e0d67d847662103ad66d1429042509d4e6ca0ec9e06ed41c1bde1c1e4/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5765c4baad6b0b8d0559cf221c7257cce01a022f8c455c4933c30594bc56af6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5765c4baad6b0b8d0559cf221c7257cce01a022f8c455c4933c30594bc56af6/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cad8a281346db8086044899f04ea993d595166fdae54ba4b72a52b998a83486a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cad8a281346db8086044899f04ea993d595166fdae54ba4b72a52b998a83486a/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3ce7db4b3804e84f9642e66b86195b7a91f1c9ac2e3dd86e63c0085bc12f480/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3ce7db4b3804e84f9642e66b86195b7a91f1c9ac2e3dd86e63c0085bc12f480/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volumes/kubernetes.io~projected/kube-api-access-xkqf5:{mountpoint:/var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volumes/kubernetes.io~projected/kube-api-access-xkqf5 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11a94da7-4353-4baf-9cd9-e585ff91ac82/volumes/kubernetes.io~projected/kube-api-access-d5lnc:{mountpoint:/var/lib/kubelet/pods/11a94da7-4353-4baf-9cd9-e585ff91ac82/volumes/kubernetes.io~projected/kube-api-access-d5lnc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13dda6c1-f004-46b6-928c-47d6de1d58f9/volumes/kubernetes.io~projected/kube-api-access-6w6wb:{mountpoint:/var/lib/kubelet/pods/13dda6c1-f004-46b6-928c-47d6de1d58f9/volumes/kubernetes.io~projected/kube-api-access-6w6wb major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/228f5775-1d78-4310-a6d4-fccb3f18a4f4/volumes/kubernetes.io~projected/kube-api-access-mcpt6:{mountpoint:/var/lib/kubelet/pods/228f5775-1d78-4310-a6d4-fccb3f18a4f4/volumes/kubernetes.io~projected/kube-api-access-mcpt6 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2964a2b0-45fb-4282-8cde-f2ef72de59c5/volumes/kubernetes.io~projected/kube-api-access-rm8gd:{mountpoint:/var/lib/kubelet/pods/2964a2b0-45fb-4282-8cde-f2ef72de59c5/volumes/kubernetes.io~projected/kube-api-access-rm8gd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a6b42d-1468-4f9b-85f3-4243a25e01ac/volumes/kubernetes.io~projected/kube-api-access-wr9f5:{mountpoint:/var/lib/kubelet/pods/40a6b42d-1468-4f9b-85f3-4243a25e01ac/volumes/kubernetes.io~projected/kube-api-access-wr9f5 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a6b42d-1468-4f9b-85f3-4243a25e01ac/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/40a6b42d-1468-4f9b-85f3-4243a25e01ac/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44a82b10-e0ed-4945-8f9a-845487bbb556/volumes/kubernetes.io~projected/kube-api-access-vnrcj:{mountpoint:/var/lib/kubelet/pods/44a82b10-e0ed-4945-8f9a-845487bbb556/volumes/kubernetes.io~projected/kube-api-access-vnrcj major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44a82b10-e0ed-4945-8f9a-845487bbb556/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/44a82b10-e0ed-4945-8f9a-845487bbb556/volumes/kubernetes.io~secret/signing-key major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/662d94e5-925a-4454-af65-8f767b48f365/volumes/kubernetes.io~projected/kube-api-access-69j74:{mountpoint:/var/lib/kubelet/pods/662d94e5-925a-4454-af65-8f767b48f365/volumes/kubernetes.io~projected/kube-api-access-69j74 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eddcaf9-30b4-44e8-bc77-24b08f234497/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8eddcaf9-30b4-44e8-bc77-24b08f234497/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a652af6-69fa-467b-abfe-bff66e3afbde/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9a652af6-69fa-467b-abfe-bff66e3afbde/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d5cb641-099e-4366-9d79-85d72a0e1a90/volumes/kubernetes.io~projected/kube-api-access-g4h2g:{mountpoint:/var/lib/kubelet/pods/9d5cb641-099e-4366-9d79-85d72a0e1a90/volumes/kubernetes.io~projected/kube-api-access-g4h2g major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139/volumes/kubernetes.io~projected/kube-api-access-klmnk:{mountpoint:/var/lib/kubelet/pods/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139/volumes/kubernetes.io~projected/kube-api-access-klmnk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~projected/kube-api-access-z9g74:{mountpoint:/var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~projected/kube-api-access-z9g74 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c020730b-79dc-43cb-9732-2102974c80c0/volumes/kubernetes.io~projected/kube-api-access-vcf4v:{mountpoint:/var/lib/kubelet/pods/c020730b-79dc-43cb-9732-2102974c80c0/volumes/kubernetes.io~projected/kube-api-access-vcf4v major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddd648a5-3a08-42c1-b8a3-e14ae414263c/volumes/kubernetes.io~projected/kube-api-access-x9bgw:{mountpoint:/var/lib/kubelet/pods/ddd648a5-3a08-42c1-b8a3-e14ae414263c/volumes/kubernetes.io~projected/kube-api-access-x9bgw major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e55a9e03-4858-42b2-a27a-a9533e404588/volumes/kubernetes.io~projected/kube-api-access-7b4gs:{mountpoint:/var/lib/kubelet/pods/e55a9e03-4858-42b2-a27a-a9533e404588/volumes/kubernetes.io~projected/kube-api-access-7b4gs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9fa46f4-4f1e-4092-a839-1dd075263b50/volumes/kubernetes.io~projected/kube-api-access-kph6l:{mountpoint:/var/lib/kubelet/pods/e9fa46f4-4f1e-4092-a839-1dd075263b50/volumes/kubernetes.io~projected/kube-api-access-kph6l major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cb24a23fabe348ec93e1d3d9c3cbc286af4ce41797f31f0697d69ae0dcd39733/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/faa4dd9972f2095077e0335f7318c391fd17103cb565252068167fdce09181a7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/53923277941b1f372e0317cfe35f54df5e3cb52524e6b4893afd281022557aa8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/19eb63fcada357f117d80933b924bea9858b9680d90e24cfe8d5c65aaec4ca4d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c3b3eb15120a42259b090b822b8b7e419feb9175fa9223665572ae82fe10aa72/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e8aa79df9800af39584c9ae6403bcaddad5d95af76066332589806fed2d30ebb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c97e274952a5136a95822f0892602d8362cfb34aed8e646b8068b788bc2ec45c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a199d251092ed865749c109c69e8b5f586d403ed9f110db801d9564c8bb4de1c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/7ed64756ca431a405b22415516d2998411007653aa7a32544f609df44328cf19/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/b570eabedda1450daf57f52290096d8af0f7d724108494334bf3f438c813e85d/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/752edf3a42e47f800c19d7819386f578ea1a8707131eb7e761dfc4f4d0a3f953/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/652913f17e2948b2648907f8426d749a76f307200387dafca90af04e26f245a9/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3ea647be7638168c67268e9f90cea4ed21730f934d254342c80d2b87e77349bb/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7a89a1c30cddb10fcc05650ef35ffe0262ffd628a59fb71e40cb96ec7f793d00/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/564ee545cb7eb9d60392409594f88eeb626784426833d6afbb555543694be16c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8ca91a5723fe9085bba8c58902fa2998cde50f869a0359b3a5f4162822fd2244/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c7f3dd5120a7fe0364d540463f42fb9e8794639062c313ee0156ef837d29a5f9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1710a133de014d3bed4147e6b9c113615c5f028882955fb1b31b057feef21033/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6bb9e1b7d44dc97859bb5ad45e9466773391a001e8f5e2ac1c5cc4473bff9274/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e6f0f11d36f533625dec5c13675333feea5b11dfcf7fdbecb390ae051daa4274/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/08cf4a6fd849abec6c1402ce3f3b8467e091b621eedd326c937a102af0500ce5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/2ba70c0b4e2162b252deb0648f3e2e6e112a9ebc715eee4e06e1eae8dfe9dd8a/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/633a887c753ee8ce67a0f365ccabb794f5502a4acf8b2a52d6df619349143178/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/e0d04d42dea79c42fe1af99c3afe53fccf78cedee4e71fee4b991166a1054d6c/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/29876bb218fbb8d30616cb614d9e67d451d7e6cca5cead4b80e8d159f77163e8/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/0f53595ac0717927ca314b0cd205d2156d1bfa7eb2a71cb5b8d1c3f3086b4ce1/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/b6c34c56a7e8ed0589161b2a3dddd2c3c5178b007410932ffe33a1cf21a38a02/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/de987eb728ede7e204315f27babdbadd74d8add3d7f8425b46b1df6d522e0624/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/fc1ebd2c9a08789397cadd410f4d3885df997923a61bc28bca390d67a435a16e/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/440882bb359f04a122de6550bb72a7cd5ccbf15e6853b85c779f4ac662d1f894/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/81dddd744a79d3c65f5e8339c28b0450dfe6d90d2bdefb466ecf4ec54889709e/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/db64aa1ec2947098e43e431b5f5427d8cb2bc7d8ad36726ced458a82eca2a9d8/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/259d19fab622683313036d1c9d0ff5521af9b7aeb853afda6381ff382e79d6d3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dc1f17644385a0c237fd2c83c4ac4a122e2af76b3f93f36ce175a53acf6052f3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/90ecfb5badbe02ec3a3099096b80e493968d66e3a42f4901dc151c39b149fb07/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0fe9e06958fee743ac1031a09c11f60c8aaf070ccdfae959f3c75d2d200ac655/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/426a76e77de537fecbddb62b732291f6a803510b22149dfeeb0b61c1f92d7426/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/f1a237400d10c7c14215f9fb09a6dc9dd4a199d05afe00e549eb1cec312b8bee/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3ca66667f56963197cfc320e481a471a03f64b16d9d853d0d12e9deeb297da74/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/821a9c8d287dd23a43a827ff0fd0f33b8b80860d24f318f579f683e9a2af6595/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.847320 4227 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:52.846818494 +0000 UTC m=+0.083796361 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3151294 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2149c9c65fe5d121502aedd321e630 SystemUUID:ec2149c9-c65f-e5d1-2150-2aedd321e630 BootID:c9ab5f11-d4ac-40d7-b758-09b6df88b8cd Filesystems:[{Device:/var/lib/kubelet/pods/228f5775-1d78-4310-a6d4-fccb3f18a4f4/volumes/kubernetes.io~projected/kube-api-access-mcpt6 DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/40a6b42d-1468-4f9b-85f3-4243a25e01ac/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cad8a281346db8086044899f04ea993d595166fdae54ba4b72a52b998a83486a/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a12e8b1edba5656cb5309eb74313ebe671e47c7871aa043e366a985ca8bf157/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12d6fa1c8cce7fb251045fd4e88e38b30b084792c3c99018abbbec3602f484c0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2828a6e0d67d847662103ad66d1429042509d4e6ca0ec9e06ed41c1bde1c1e4/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~projected/kube-api-access-z9g74 DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139/volumes/kubernetes.io~projected/kube-api-access-klmnk DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f0b001be9f8130d7f4d6fd8eea51504a326ee41258e2dd23539a960934cd85e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41f236f00d4432250b764f7c2e71d99483e2036dab2684cbb0a9ecc4de46c278/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a369a7f73a85eef1737426f905fc9b1695ba79250be6e1db369d4c08e80d141/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/680a8b70eddda0b83d0621e15bf125399f531be96ce3e724bc3f79b7df1df697/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40a6b42d-1468-4f9b-85f3-4243a25e01ac/volumes/kubernetes.io~projected/kube-api-access-wr9f5 DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2964a2b0-45fb-4282-8cde-f2ef72de59c5/volumes/kubernetes.io~projected/kube-api-access-rm8gd DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3ce7db4b3804e84f9642e66b86195b7a91f1c9ac2e3dd86e63c0085bc12f480/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddd648a5-3a08-42c1-b8a3-e14ae414263c/volumes/kubernetes.io~projected/kube-api-access-x9bgw DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8eddcaf9-30b4-44e8-bc77-24b08f234497/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/11a94da7-4353-4baf-9cd9-e585ff91ac82/volumes/kubernetes.io~projected/kube-api-access-d5lnc DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9d5cb641-099e-4366-9d79-85d72a0e1a90/volumes/kubernetes.io~projected/kube-api-access-g4h2g DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44a82b10-e0ed-4945-8f9a-845487bbb556/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a652af6-69fa-467b-abfe-bff66e3afbde/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61d8f3d671acb9b9c0340abfe614b18c7cdad38c243dc3461348d27fe6fc2d59/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e55a9e03-4858-42b2-a27a-a9533e404588/volumes/kubernetes.io~projected/kube-api-access-7b4gs DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/662d94e5-925a-4454-af65-8f767b48f365/volumes/kubernetes.io~projected/kube-api-access-69j74 DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9fa46f4-4f1e-4092-a839-1dd075263b50/volumes/kubernetes.io~projected/kube-api-access-kph6l DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5765c4baad6b0b8d0559cf221c7257cce01a022f8c455c4933c30594bc56af6/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/13dda6c1-f004-46b6-928c-47d6de1d58f9/volumes/kubernetes.io~projected/kube-api-access-6w6wb DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ed7e4eb9858d00756e506026c660a1d367c159e38c375d1a78fe709a4ba8a68/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/0568938d-c9e2-4e37-b768-cdeddaacf33e/volumes/kubernetes.io~projected/kube-api-access-xkqf5 DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44a82b10-e0ed-4945-8f9a-845487bbb556/volumes/kubernetes.io~projected/kube-api-access-vnrcj DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c020730b-79dc-43cb-9732-2102974c80c0/volumes/kubernetes.io~projected/kube-api-access-vcf4v DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51b0b83934216acaba00407086965a33988662f7d94c03bd74888c594ada8a13/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba768925-3dbc-4dcb-a35c-54e20f5bf36b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:51b0b83934216ac MacAddress:ba:ff:cb:aa:89:4f Speed:10000 Mtu:8901} {Name:61d8f3d671acb9b MacAddress:06:33:e6:0c:4a:de Speed:10000 Mtu:8901} {Name:680a8b70eddda0b MacAddress:0e:7c:95:b9:54:af Speed:10000 Mtu:8901} {Name:a2828a6e0d67d84 MacAddress:fe:52:35:83:7c:f4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d1:e5:2b:29 Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:db:d3:fa:90:49 Speed:0 Mtu:8901} {Name:d3ce7db4b3804e8 MacAddress:82:86:f1:5f:18:d4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d1:e5:2b:29 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:aa:dd:be:39:d3:fb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:76:fd:67:8a:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.847576 4227 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.847676 4227 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.847971 4227 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.847996 4227 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-71.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848116 4227 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848123 4227 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848134 4227 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848146 4227 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848224 4227 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848413 4227 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848513 4227 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848544 4227 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848556 4227 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848565 4227 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848574 4227 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.848581 4227 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.849110 4227 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.849122 4227 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:52.852816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.849578 4227 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:28:52.856704 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.856690 4227 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:28:52.856704 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.856703 4227 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:52.856803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.856794 4227 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:28:52.856971 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.856962 4227 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:28:52.857001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.856990 4227 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:28:52.857079 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857070 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:28:52.857111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857084 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:28:52.857111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857090 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:28:52.857111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857096 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:28:52.857111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857102 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:28:52.857111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857107 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857113 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857119 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857125 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857131 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857139 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857148 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857189 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:28:52.857269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.857196 4227 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:28:52.860237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.860224 4227 server.go:1263] "Started kubelet" Nov 03 20:28:52.860318 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.860291 4227 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:28:52.860387 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.860313 4227 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:28:52.860811 ip-10-0-134-71 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:28:52.861603 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.860849 4227 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:28:52.861603 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.861121 4227 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:28:52.864422 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.864400 4227 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:28:52.865355 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.865338 4227 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:28:52.865744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.865385 4227 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:52.865744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.865427 4227 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:13 +0000 UTC" deadline="2028-06-30 10:17:16.921373701 +0000 UTC" Nov 03 20:28:52.865744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.865444 4227 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14509h48m24.055933003s" Nov 03 20:28:52.865744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.865497 4227 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:28:52.865744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.865507 4227 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:28:52.865744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.865592 4227 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:28:52.865744 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:52.865645 4227 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" Nov 03 20:28:52.867192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867142 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0568938d-c9e2-4e37-b768-cdeddaacf33e" volumeName="kubernetes.io/secret/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:28:52.867233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867199 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13dda6c1-f004-46b6-928c-47d6de1d58f9" volumeName="kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-binary-copy" seLinuxMountContext="" Nov 03 20:28:52.867233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867214 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd648a5-3a08-42c1-b8a3-e14ae414263c" volumeName="kubernetes.io/empty-dir/ddd648a5-3a08-42c1-b8a3-e14ae414263c-data-volume" seLinuxMountContext="" Nov 03 20:28:52.867233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867228 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd648a5-3a08-42c1-b8a3-e14ae414263c" volumeName="kubernetes.io/projected/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-api-access-x9bgw" seLinuxMountContext="" Nov 03 20:28:52.867308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867241 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e55a9e03-4858-42b2-a27a-a9533e404588" volumeName="kubernetes.io/projected/e55a9e03-4858-42b2-a27a-a9533e404588-kube-api-access-7b4gs" seLinuxMountContext="" Nov 03 20:28:52.867308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867254 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0568938d-c9e2-4e37-b768-cdeddaacf33e" volumeName="kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:28:52.867308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867266 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="228f5775-1d78-4310-a6d4-fccb3f18a4f4" volumeName="kubernetes.io/projected/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kube-api-access-mcpt6" seLinuxMountContext="" Nov 03 20:28:52.867308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867277 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a6b42d-1468-4f9b-85f3-4243a25e01ac" volumeName="kubernetes.io/projected/40a6b42d-1468-4f9b-85f3-4243a25e01ac-kube-api-access-wr9f5" seLinuxMountContext="" Nov 03 20:28:52.867308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867292 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44a82b10-e0ed-4945-8f9a-845487bbb556" volumeName="kubernetes.io/configmap/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-cabundle" seLinuxMountContext="" Nov 03 20:28:52.867308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867303 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="662d94e5-925a-4454-af65-8f767b48f365" volumeName="kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867314 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa839ba6-7b29-41c8-bd50-a7cd5b1ac139" volumeName="kubernetes.io/projected/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-kube-api-access-klmnk" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867326 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c020730b-79dc-43cb-9732-2102974c80c0" volumeName="kubernetes.io/projected/c020730b-79dc-43cb-9732-2102974c80c0-kube-api-access-vcf4v" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867339 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11a94da7-4353-4baf-9cd9-e585ff91ac82" volumeName="kubernetes.io/configmap/11a94da7-4353-4baf-9cd9-e585ff91ac82-serviceca" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867349 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba768925-3dbc-4dcb-a35c-54e20f5bf36b" volumeName="kubernetes.io/projected/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-kube-api-access-z9g74" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867364 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5cb641-099e-4366-9d79-85d72a0e1a90" volumeName="kubernetes.io/projected/9d5cb641-099e-4366-9d79-85d72a0e1a90-kube-api-access-g4h2g" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867374 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0568938d-c9e2-4e37-b768-cdeddaacf33e" volumeName="kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-config" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867386 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0568938d-c9e2-4e37-b768-cdeddaacf33e" volumeName="kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-env-overrides" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867398 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11a94da7-4353-4baf-9cd9-e585ff91ac82" volumeName="kubernetes.io/projected/11a94da7-4353-4baf-9cd9-e585ff91ac82-kube-api-access-d5lnc" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867412 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2964a2b0-45fb-4282-8cde-f2ef72de59c5" volumeName="kubernetes.io/empty-dir/2964a2b0-45fb-4282-8cde-f2ef72de59c5-tmp-dir" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867425 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2964a2b0-45fb-4282-8cde-f2ef72de59c5" volumeName="kubernetes.io/projected/2964a2b0-45fb-4282-8cde-f2ef72de59c5-kube-api-access-rm8gd" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867448 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44a82b10-e0ed-4945-8f9a-845487bbb556" volumeName="kubernetes.io/projected/44a82b10-e0ed-4945-8f9a-845487bbb556-kube-api-access-vnrcj" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867460 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd648a5-3a08-42c1-b8a3-e14ae414263c" volumeName="kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 20:28:52.867477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867471 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13dda6c1-f004-46b6-928c-47d6de1d58f9" volumeName="kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867484 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2964a2b0-45fb-4282-8cde-f2ef72de59c5" volumeName="kubernetes.io/configmap/2964a2b0-45fb-4282-8cde-f2ef72de59c5-config-volume" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867509 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9fa46f4-4f1e-4092-a839-1dd075263b50" volumeName="kubernetes.io/configmap/e9fa46f4-4f1e-4092-a839-1dd075263b50-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867521 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9fa46f4-4f1e-4092-a839-1dd075263b50" volumeName="kubernetes.io/projected/e9fa46f4-4f1e-4092-a839-1dd075263b50-kube-api-access-kph6l" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867533 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13dda6c1-f004-46b6-928c-47d6de1d58f9" volumeName="kubernetes.io/projected/13dda6c1-f004-46b6-928c-47d6de1d58f9-kube-api-access-6w6wb" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867547 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a6b42d-1468-4f9b-85f3-4243a25e01ac" volumeName="kubernetes.io/secret/40a6b42d-1468-4f9b-85f3-4243a25e01ac-cert" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867566 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a652af6-69fa-467b-abfe-bff66e3afbde" volumeName="kubernetes.io/secret/9a652af6-69fa-467b-abfe-bff66e3afbde-agent-certs" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867578 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5cb641-099e-4366-9d79-85d72a0e1a90" volumeName="kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-daemon-config" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867590 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13dda6c1-f004-46b6-928c-47d6de1d58f9" volumeName="kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867601 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44a82b10-e0ed-4945-8f9a-845487bbb556" volumeName="kubernetes.io/secret/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-key" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867614 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eddcaf9-30b4-44e8-bc77-24b08f234497" volumeName="kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867627 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa839ba6-7b29-41c8-bd50-a7cd5b1ac139" volumeName="kubernetes.io/empty-dir/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-tmp-dir" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867638 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba768925-3dbc-4dcb-a35c-54e20f5bf36b" volumeName="kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-tmp" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867649 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5cb641-099e-4366-9d79-85d72a0e1a90" volumeName="kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-cni-binary-copy" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867661 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0568938d-c9e2-4e37-b768-cdeddaacf33e" volumeName="kubernetes.io/projected/0568938d-c9e2-4e37-b768-cdeddaacf33e-kube-api-access-xkqf5" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867673 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a652af6-69fa-467b-abfe-bff66e3afbde" volumeName="kubernetes.io/configmap/9a652af6-69fa-467b-abfe-bff66e3afbde-konnectivity-ca" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867687 4227 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba768925-3dbc-4dcb-a35c-54e20f5bf36b" volumeName="kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-tuned" seLinuxMountContext="" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867700 4227 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:28:52.867855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.867706 4227 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:28:52.868611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.868523 4227 factory.go:153] Registering CRI-O factory Nov 03 20:28:52.868611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.868539 4227 factory.go:223] Registration of the crio container factory successfully Nov 03 20:28:52.868611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.868592 4227 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:28:52.868611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.868603 4227 factory.go:55] Registering systemd factory Nov 03 20:28:52.868611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.868611 4227 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:28:52.868772 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.868628 4227 factory.go:103] Registering Raw factory Nov 03 20:28:52.868772 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.868642 4227 manager.go:1199] Started watching for new ooms in manager Nov 03 20:28:52.869032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.869008 4227 manager.go:319] Starting recovery of all containers Nov 03 20:28:52.872886 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.872870 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:28:52.874277 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.874261 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:28:52.890934 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.890903 4227 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:28:52.892230 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.892209 4227 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:28:52.892295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.892234 4227 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:28:52.892295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.892254 4227 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:28:52.892401 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:52.892293 4227 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:28:52.921117 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.921098 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:28:52.921526 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.921491 4227 generic.go:356] "Generic (PLEG): container finished" podID="0568938d-c9e2-4e37-b768-cdeddaacf33e" containerID="888636e7250d02747d4bac990725ab910abdcb32e778d575cd66c10dee803b84" exitCode=1 Nov 03 20:28:52.925038 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.925016 4227 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="a0f20dbaac3831695a857409114692d77ba856015b38986ccf74fbb32c3518c5" exitCode=0 Nov 03 20:28:52.925127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.925039 4227 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="80556bba002a969a92a0ab0b446101f9e574da426fda410e3fb2551e462adb91" exitCode=0 Nov 03 20:28:52.925127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.925051 4227 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="22dd297b5c5eb550a9e859e0d6a16a4a68d35546e1957d9df1c23d86edb9dedf" exitCode=0 Nov 03 20:28:52.925127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.925059 4227 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="a2355025b14a066252a542c6a0d323e43f362541f4a94a28f1ce38c9323234b5" exitCode=0 Nov 03 20:28:52.925127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.925066 4227 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="75f7ce5f75a99f9e4164cddafa6f434ba1bb64540c02a297d4d6c117e62a5951" exitCode=0 Nov 03 20:28:52.925127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.925076 4227 generic.go:356] "Generic (PLEG): container finished" podID="13dda6c1-f004-46b6-928c-47d6de1d58f9" containerID="2c7c5e0f4dd1b66c9aaf5abefbf1dddc3aad7b91f6b7513583138d15e9918095" exitCode=0 Nov 03 20:28:52.926942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.926889 4227 generic.go:356] "Generic (PLEG): container finished" podID="65c24ecfbcdfa9586bb1da6fb1e34a7c" containerID="0a86cabacfe43ca317860846cb27d734061f0f05344b77368c68076e125a8d88" exitCode=0 Nov 03 20:28:52.935375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.935257 4227 manager.go:324] Recovery completed Nov 03 20:28:52.946950 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.946933 4227 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:52.947965 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.947951 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:52.948025 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.947977 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:52.948025 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.947987 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:52.949267 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949253 4227 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:28:52.949267 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949263 4227 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:52.949350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949265 4227 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:28:52.949350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949332 4227 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:28:52.949480 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949466 4227 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:28:52.949529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949480 4227 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:28:52.949529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949495 4227 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:28:52.949529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949501 4227 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:28:52.949529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949506 4227 policy_none.go:50] "Start" Nov 03 20:28:52.949529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949515 4227 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:28:52.949529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949525 4227 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:28:52.949696 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949636 4227 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:28:52.949696 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949643 4227 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:28:52.949696 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.949648 4227 policy_none.go:44] "Start" Nov 03 20:28:52.950050 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.950030 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:52.950147 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.950060 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:52.950147 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.950074 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:52.954078 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.954039 4227 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:28:52.956598 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.956583 4227 manager.go:341] "Starting Device Plugin manager" Nov 03 20:28:52.956698 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:52.956613 4227 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:28:52.956698 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.956625 4227 server.go:87] "Starting device plugin registration server" Nov 03 20:28:52.956889 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.956875 4227 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:28:52.956949 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.956890 4227 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:28:52.957003 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.956990 4227 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:28:52.957115 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.957080 4227 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:28:52.957205 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.957121 4227 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:28:52.957205 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.957127 4227 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:28:52.957300 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.957209 4227 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:28:52.957111958Z","UUID":"9bac9d51-5363-4c04-892c-0f8b64661285","Handler":null,"Name":"","Endpoint":""} Nov 03 20:28:52.958366 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:52.958324 4227 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:28:52.958366 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:52.958363 4227 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" Nov 03 20:28:52.958546 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.958532 4227 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:28:52.958620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.958586 4227 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:28:52.993426 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.993390 4227 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal"] Nov 03 20:28:52.993508 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.993498 4227 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:52.994421 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.994405 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:52.994486 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.994430 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:52.994486 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.994440 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:52.994652 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.994642 4227 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:52.994797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.994780 4227 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:52.995452 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995437 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:52.995551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995462 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:52.995551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995465 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:52.995551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995476 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:52.995551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995484 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:52.995551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995494 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:52.995754 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:52.995632 4227 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:52.995754 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995687 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"b295d4bbd28e0091df21418ae66e44d9","Type":"ContainerStarted","Data":"870cde1061065317517a9fb74f059bd9ea7fc6c0c0303eaee0d4e6831db95d31"} Nov 03 20:28:52.995754 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995716 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"b295d4bbd28e0091df21418ae66e44d9","Type":"ContainerStarted","Data":"f249ee0adc57758ca6fe55c6304c758553eb7f1d8baac51bf354b8d8265abf6b"} Nov 03 20:28:52.995841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995773 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"65c24ecfbcdfa9586bb1da6fb1e34a7c","Type":"ContainerStarted","Data":"05255c862fc64996d4a7a02ec2b1e690a39e89edd6e673117ff3759dcab6fbfe"} Nov 03 20:28:52.995841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995785 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"65c24ecfbcdfa9586bb1da6fb1e34a7c","Type":"ContainerDied","Data":"0a86cabacfe43ca317860846cb27d734061f0f05344b77368c68076e125a8d88"} Nov 03 20:28:52.995841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995796 4227 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:52.995841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.995796 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" event={"ID":"65c24ecfbcdfa9586bb1da6fb1e34a7c","Type":"ContainerStarted","Data":"eb683299a76b641749dbd8c2a8e91f1d26c95c29e9363a876b0173b08780e133"} Nov 03 20:28:52.996579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.996563 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:52.996644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.996588 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:52.996644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:52.996598 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:52.996721 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:52.996704 4227 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-71.us-west-2.compute.internal\" not found" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.041783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.041762 4227 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:53.041783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.041774 4227 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:53.041904 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.041863 4227 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:28:53.042865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.042849 4227 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:53.056999 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.056982 4227 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:53.057742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.057726 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:53.057814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.057755 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:53.057814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.057765 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:53.057814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.057798 4227 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.065877 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.065865 4227 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.089481 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.089459 4227 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:53.133856 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.133836 4227 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:53.133936 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:53.133878 4227 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.133936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.133892 4227 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.142506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.142485 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.142581 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.142510 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.142581 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.142527 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b295d4bbd28e0091df21418ae66e44d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"b295d4bbd28e0091df21418ae66e44d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.159756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.159735 4227 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.159830 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.159790 4227 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.160207 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.160191 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeNotReady" Nov 03 20:28:53.160265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.160220 4227 setters.go:546] "Node became not ready" node="ip-10-0-134-71.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T20:28:53Z","lastTransitionTime":"2026-11-03T20:28:53Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 20:28:53.185734 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.185688 4227 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:53.185734 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:53.185721 4227 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.191859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.191841 4227 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-71.us-west-2.compute.internal" is owned by different node (oldNodeID="02a39398-0b0d-4d7e-8fb5-77e0669e5eec", newNodeID=""), cleaning up... Nov 03 20:28:53.243239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.243217 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.243306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.243239 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b295d4bbd28e0091df21418ae66e44d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"b295d4bbd28e0091df21418ae66e44d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.243306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.243255 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.243306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.243290 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.243401 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.243330 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b295d4bbd28e0091df21418ae66e44d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"b295d4bbd28e0091df21418ae66e44d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.243401 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.243330 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c24ecfbcdfa9586bb1da6fb1e34a7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" (UID: \"65c24ecfbcdfa9586bb1da6fb1e34a7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.848901 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.848881 4227 apiserver.go:52] "Watching apiserver" Nov 03 20:28:53.929525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.929504 4227 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.945797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.945778 4227 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:28:53.946214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.946196 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-xx8t5","openshift-dns/node-resolver-4wjxm","openshift-multus/multus-additional-cni-plugins-jr82s","openshift-network-operator/iptables-alerter-gcz42","kube-system/global-pull-secret-syncer-xfqt9","openshift-dns/dns-default-9k29p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc","openshift-insights/insights-runtime-extractor-wcndc","kube-system/konnectivity-agent-hc7pp","openshift-image-registry/node-ca-dg8z2","openshift-cluster-node-tuning-operator/tuned-92trd","openshift-multus/network-metrics-daemon-xqvnn","kube-system/kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-7v5wf","openshift-service-ca/service-ca-5fffc97bdf-8mp2t","openshift-ingress-canary/ingress-canary-lrdnf","openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n","openshift-multus/multus-wwvxj"] Nov 03 20:28:53.951770 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.951750 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:53.953458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.953442 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:53.953819 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.953800 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:53.967278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.967261 4227 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:28:53.984464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.984451 4227 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:53.984541 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:53.984489 4227 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-71.us-west-2.compute.internal" Nov 03 20:28:53.996566 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.996352 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d54s8\"" Nov 03 20:28:53.996566 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.996396 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:28:53.996566 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.996404 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:28:53.996566 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.996407 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mlgd9\"" Nov 03 20:28:53.996566 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.996460 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:28:53.996771 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.996616 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:28:53.998253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:53.998241 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jb4cj\"" Nov 03 20:28:54.045956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.045937 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-kubelet\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.046003 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.045963 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-config\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.046003 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.045980 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqf5\" (UniqueName: \"kubernetes.io/projected/0568938d-c9e2-4e37-b768-cdeddaacf33e-kube-api-access-xkqf5\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.046003 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.045995 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.046119 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046030 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-var-lib-kubelet\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.046119 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046045 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.046273 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046255 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-key\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:54.046329 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046309 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-run\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.046397 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046384 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8gd\" (UniqueName: \"kubernetes.io/projected/2964a2b0-45fb-4282-8cde-f2ef72de59c5-kube-api-access-rm8gd\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.046429 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046415 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9fa46f4-4f1e-4092-a839-1dd075263b50-iptables-alerter-script\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:54.046502 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046486 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-k8s-cni-cncf-io\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.046532 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046518 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-multus-certs\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.046575 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046554 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11a94da7-4353-4baf-9cd9-e585ff91ac82-serviceca\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:54.046614 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046589 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lnc\" (UniqueName: \"kubernetes.io/projected/11a94da7-4353-4baf-9cd9-e585ff91ac82-kube-api-access-d5lnc\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:54.046656 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046642 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.046691 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046663 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.046853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046700 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.046853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046730 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.046853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046762 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-cabundle\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:54.046853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046782 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-etc-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.046853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046797 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-kubernetes\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.046853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046815 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-sys\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.046853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046850 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph6l\" (UniqueName: \"kubernetes.io/projected/e9fa46f4-4f1e-4092-a839-1dd075263b50-kube-api-access-kph6l\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046885 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046920 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-node-log\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046947 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovn-node-metrics-cert\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.046974 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bgw\" (UniqueName: \"kubernetes.io/projected/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-api-access-x9bgw\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047001 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-conf\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047036 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4gs\" (UniqueName: \"kubernetes.io/projected/e55a9e03-4858-42b2-a27a-a9533e404588-kube-api-access-7b4gs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047126 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-hosts-file\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047168 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd648a5-3a08-42c1-b8a3-e14ae414263c-crio-socket\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.047199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047187 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd648a5-3a08-42c1-b8a3-e14ae414263c-data-volume\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047220 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047248 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047251 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd648a5-3a08-42c1-b8a3-e14ae414263c-data-volume\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047291 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-tuned\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047320 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2964a2b0-45fb-4282-8cde-f2ef72de59c5-config-volume\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047350 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmnk\" (UniqueName: \"kubernetes.io/projected/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-kube-api-access-klmnk\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047356 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-tuned\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047389 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-slash\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047413 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-lib-modules\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047439 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-tmp\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047470 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fa46f4-4f1e-4092-a839-1dd075263b50-host-slash\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047506 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047529 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-tmp\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047538 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-modprobe-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.047606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047573 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047633 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047662 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047691 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcpt6\" (UniqueName: \"kubernetes.io/projected/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kube-api-access-mcpt6\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047717 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047744 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcf4v\" (UniqueName: \"kubernetes.io/projected/c020730b-79dc-43cb-9732-2102974c80c0-kube-api-access-vcf4v\") pod \"migrator-66df5d496b-59x5n\" (UID: \"c020730b-79dc-43cb-9732-2102974c80c0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047825 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047891 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4h2g\" (UniqueName: \"kubernetes.io/projected/9d5cb641-099e-4366-9d79-85d72a0e1a90-kube-api-access-g4h2g\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047909 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047924 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-systemd-units\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047939 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-script-lib\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.047955 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-hostroot\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048000 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-etc-kubernetes\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048064 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-ovn\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048132 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-env-overrides\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048151 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-host\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048203 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.048250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048234 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-cni-binary-copy\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048269 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a652af6-69fa-467b-abfe-bff66e3afbde-konnectivity-ca\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048297 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40a6b42d-1468-4f9b-85f3-4243a25e01ac-cert\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048331 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-netns\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048350 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-var-lib-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048376 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-netd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048418 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g74\" (UniqueName: \"kubernetes.io/projected/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-kube-api-access-z9g74\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048509 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048624 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-cnibin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048661 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-daemon-config\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048655 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a652af6-69fa-467b-abfe-bff66e3afbde-konnectivity-ca\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048703 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-systemd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048756 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-log-socket\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048801 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-socket-dir-parent\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048844 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-conf-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048869 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a652af6-69fa-467b-abfe-bff66e3afbde-agent-certs\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:54.048936 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048930 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-kubelet-config\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.048955 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-dbus\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049002 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-tmp-dir\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049054 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w6wb\" (UniqueName: \"kubernetes.io/projected/13dda6c1-f004-46b6-928c-47d6de1d58f9-kube-api-access-6w6wb\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049092 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-tmp-dir\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049115 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049224 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a652af6-69fa-467b-abfe-bff66e3afbde-agent-certs\") pod \"konnectivity-agent-hc7pp\" (UID: \"9a652af6-69fa-467b-abfe-bff66e3afbde\") " pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049227 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-netns\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049271 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-bin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049305 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-multus\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049333 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-kubelet\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049358 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049375 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2964a2b0-45fb-4282-8cde-f2ef72de59c5-tmp-dir\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049413 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9f5\" (UniqueName: \"kubernetes.io/projected/40a6b42d-1468-4f9b-85f3-4243a25e01ac-kube-api-access-wr9f5\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049432 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2964a2b0-45fb-4282-8cde-f2ef72de59c5-tmp-dir\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049465 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-cnibin\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049489 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049505 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-bin\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.049562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049556 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysconfig\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.050121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049580 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-systemd\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.050121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049597 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-system-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.050121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049646 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a94da7-4353-4baf-9cd9-e585ff91ac82-host\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:54.050121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049681 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrcj\" (UniqueName: \"kubernetes.io/projected/44a82b10-e0ed-4945-8f9a-845487bbb556-kube-api-access-vnrcj\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:54.050121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.049751 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-system-cni-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.096849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.096823 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.097476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.097357 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.097679 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.097641 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.097986 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.097970 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:28:54.098120 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098072 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:28:54.098120 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098093 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:28:54.098251 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098135 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:28:54.098251 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098216 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxhx8\"" Nov 03 20:28:54.098465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098449 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:28:54.098516 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098459 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:28:54.098516 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098479 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.098596 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098510 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.098596 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098518 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.098596 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098578 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.098729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098630 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.098729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098635 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.098729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098631 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.098729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098648 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:28:54.098729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098674 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.098729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098706 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098735 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098781 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098822 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098832 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098842 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098874 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098872 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.098943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098903 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098956 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6vtd\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.098980 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099016 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099053 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6582\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099077 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099103 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099132 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099137 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099147 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099213 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099232 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:28:54.099254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099254 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:54.099619 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099263 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:28:54.099619 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099310 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:28:54.099619 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099401 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w48hg\"" Nov 03 20:28:54.099619 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099447 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:28:54.099722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.099676 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h9ssl\"" Nov 03 20:28:54.106861 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.106835 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11a94da7-4353-4baf-9cd9-e585ff91ac82-serviceca\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:54.106861 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.106857 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9fa46f4-4f1e-4092-a839-1dd075263b50-iptables-alerter-script\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:54.106991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.106873 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-key\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:54.106991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.106838 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-config\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.106991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.106961 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.107144 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.107107 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.107210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.107150 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovn-node-metrics-cert\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.107210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.107151 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/44a82b10-e0ed-4945-8f9a-845487bbb556-signing-cabundle\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:54.108065 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.108048 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2964a2b0-45fb-4282-8cde-f2ef72de59c5-config-volume\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.108129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.108057 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.108246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.108229 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13dda6c1-f004-46b6-928c-47d6de1d58f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.108388 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.108360 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-env-overrides\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.108507 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.108494 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-cni-binary-copy\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.108571 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.108558 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40a6b42d-1468-4f9b-85f3-4243a25e01ac-cert\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:54.109192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.109178 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-daemon-config\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.141682 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.141659 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntq9n\"" Nov 03 20:28:54.141763 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.141683 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kpkvl\"" Nov 03 20:28:54.141854 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.141841 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fpmzx\"" Nov 03 20:28:54.141927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.141913 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pqh8h\"" Nov 03 20:28:54.141974 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.141913 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2t6xn\"" Nov 03 20:28:54.142031 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.141985 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8hb94\"" Nov 03 20:28:54.142102 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142086 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9qmc\"" Nov 03 20:28:54.142151 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142115 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:28:54.142216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142146 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8pktm\"" Nov 03 20:28:54.142272 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142212 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:28:54.142272 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142223 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jxjwz\"" Nov 03 20:28:54.142708 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142682 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bgw\" (UniqueName: \"kubernetes.io/projected/ddd648a5-3a08-42c1-b8a3-e14ae414263c-kube-api-access-x9bgw\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.142797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142781 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8gd\" (UniqueName: \"kubernetes.io/projected/2964a2b0-45fb-4282-8cde-f2ef72de59c5-kube-api-access-rm8gd\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.142797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142788 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph6l\" (UniqueName: \"kubernetes.io/projected/e9fa46f4-4f1e-4092-a839-1dd075263b50-kube-api-access-kph6l\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:54.142891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142806 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4h2g\" (UniqueName: \"kubernetes.io/projected/9d5cb641-099e-4366-9d79-85d72a0e1a90-kube-api-access-g4h2g\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.142891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142849 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lnc\" (UniqueName: \"kubernetes.io/projected/11a94da7-4353-4baf-9cd9-e585ff91ac82-kube-api-access-d5lnc\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:54.142891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.142862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcpt6\" (UniqueName: \"kubernetes.io/projected/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kube-api-access-mcpt6\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143654 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcf4v\" (UniqueName: \"kubernetes.io/projected/c020730b-79dc-43cb-9732-2102974c80c0-kube-api-access-vcf4v\") pod \"migrator-66df5d496b-59x5n\" (UID: \"c020730b-79dc-43cb-9732-2102974c80c0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-59x5n" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143706 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g74\" (UniqueName: \"kubernetes.io/projected/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-kube-api-access-z9g74\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143768 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9f5\" (UniqueName: \"kubernetes.io/projected/40a6b42d-1468-4f9b-85f3-4243a25e01ac-kube-api-access-wr9f5\") pod \"ingress-canary-lrdnf\" (UID: \"40a6b42d-1468-4f9b-85f3-4243a25e01ac\") " pod="openshift-ingress-canary/ingress-canary-lrdnf" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143810 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4gs\" (UniqueName: \"kubernetes.io/projected/e55a9e03-4858-42b2-a27a-a9533e404588-kube-api-access-7b4gs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143811 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmnk\" (UniqueName: \"kubernetes.io/projected/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-kube-api-access-klmnk\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143947 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w6wb\" (UniqueName: \"kubernetes.io/projected/13dda6c1-f004-46b6-928c-47d6de1d58f9-kube-api-access-6w6wb\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143956 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqf5\" (UniqueName: \"kubernetes.io/projected/0568938d-c9e2-4e37-b768-cdeddaacf33e-kube-api-access-xkqf5\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.143993 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrcj\" (UniqueName: \"kubernetes.io/projected/44a82b10-e0ed-4945-8f9a-845487bbb556-kube-api-access-vnrcj\") pod \"service-ca-5fffc97bdf-8mp2t\" (UID: \"44a82b10-e0ed-4945-8f9a-845487bbb556\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" Nov 03 20:28:54.146472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.144554 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69j74\" (UniqueName: \"kubernetes.io/projected/662d94e5-925a-4454-af65-8f767b48f365-kube-api-access-69j74\") pod \"network-check-target-7v5wf\" (UID: \"662d94e5-925a-4454-af65-8f767b48f365\") " pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:28:54.147414 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.147397 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eddcaf9-30b4-44e8-bc77-24b08f234497-original-pull-secret\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.148706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.148690 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0568938d-c9e2-4e37-b768-cdeddaacf33e-ovnkube-script-lib\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150845 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-netns\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150895 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150866 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-var-lib-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150895 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150880 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-netd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150894 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150908 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-cnibin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150941 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-var-lib-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150951 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-netd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150952 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-systemd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150981 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-cnibin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150987 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-netns\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.150996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.150990 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-log-socket\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151012 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-systemd\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151017 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-socket-dir-parent\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151025 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151041 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-conf-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151046 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-log-socket\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151065 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-kubelet-config\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151071 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-socket-dir-parent\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151074 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-multus-conf-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151088 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-dbus\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151113 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151135 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-netns\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151171 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-kubelet-config\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151173 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-bin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151179 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151193 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eddcaf9-30b4-44e8-bc77-24b08f234497-dbus\") pod \"global-pull-secret-syncer-xfqt9\" (UID: \"8eddcaf9-30b4-44e8-bc77-24b08f234497\") " pod="kube-system/global-pull-secret-syncer-xfqt9" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151202 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-netns\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151221 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-bin\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151225 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-multus\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151250 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-kubelet\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151265 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151268 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-cni-multus\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151293 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-var-lib-kubelet\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.151324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151304 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151327 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-cnibin\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151360 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151375 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-bin\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151390 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysconfig\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151404 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-systemd\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151407 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151438 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-cnibin\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151445 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-cni-bin\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151454 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-systemd\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151425 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-system-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151477 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysconfig\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151481 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-system-cni-dir\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151484 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a94da7-4353-4baf-9cd9-e585ff91ac82-host\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151501 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a94da7-4353-4baf-9cd9-e585ff91ac82-host\") pod \"node-ca-dg8z2\" (UID: \"11a94da7-4353-4baf-9cd9-e585ff91ac82\") " pod="openshift-image-registry/node-ca-dg8z2" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151523 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-system-cni-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151546 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-kubelet\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151571 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151597 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-system-cni-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151593 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-var-lib-kubelet\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151600 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-kubelet\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151633 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151659 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151660 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-run\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151686 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-var-lib-kubelet\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151691 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151705 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-run\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151712 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-k8s-cni-cncf-io\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151731 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-multus-certs\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151744 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-k8s-cni-cncf-io\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151746 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151766 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151778 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13dda6c1-f004-46b6-928c-47d6de1d58f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr82s\" (UID: \"13dda6c1-f004-46b6-928c-47d6de1d58f9\") " pod="openshift-multus/multus-additional-cni-plugins-jr82s" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151780 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-etc-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151797 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-etc-openvswitch\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151803 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-kubernetes\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151814 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-host-run-multus-certs\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151825 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-kubernetes\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151818 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-sys\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151839 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-sys\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151853 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151853 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-node-log\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151886 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-conf\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151909 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-hosts-file\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151923 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-node-log\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151938 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd648a5-3a08-42c1-b8a3-e14ae414263c-crio-socket\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151947 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa839ba6-7b29-41c8-bd50-a7cd5b1ac139-hosts-file\") pod \"node-resolver-4wjxm\" (UID: \"aa839ba6-7b29-41c8-bd50-a7cd5b1ac139\") " pod="openshift-dns/node-resolver-4wjxm" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151970 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151993 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd648a5-3a08-42c1-b8a3-e14ae414263c-crio-socket\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151997 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-sysctl-conf\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.151999 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-slash\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152032 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-lib-modules\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152047 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fa46f4-4f1e-4092-a839-1dd075263b50-host-slash\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.152055 4227 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152060 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152066 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-slash\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152081 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-modprobe-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152092 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152104 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152107 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fa46f4-4f1e-4092-a839-1dd075263b50-host-slash\") pod \"iptables-alerter-gcz42\" (UID: \"e9fa46f4-4f1e-4092-a839-1dd075263b50\") " pod="openshift-network-operator/iptables-alerter-gcz42" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.152130 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:54.65210243 +0000 UTC m=+1.889080301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152151 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152154 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-etc-modprobe-d\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152173 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152212 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152218 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152224 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-lib-modules\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152236 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:54.152216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152263 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228f5775-1d78-4310-a6d4-fccb3f18a4f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7wqc\" (UID: \"228f5775-1d78-4310-a6d4-fccb3f18a4f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7wqc" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152280 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-systemd-units\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152295 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-hostroot\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152308 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-etc-kubernetes\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.152313 4227 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152322 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-ovn\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152331 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-systemd-units\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152335 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-host\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152362 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba768925-3dbc-4dcb-a35c-54e20f5bf36b-host\") pod \"tuned-92trd\" (UID: \"ba768925-3dbc-4dcb-a35c-54e20f5bf36b\") " pod="openshift-cluster-node-tuning-operator/tuned-92trd" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152377 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152377 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-etc-kubernetes\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.152390 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:54.65237443 +0000 UTC m=+1.889352288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : secret "metrics-daemon-secret" not found Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152392 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0568938d-c9e2-4e37-b768-cdeddaacf33e-run-ovn\") pod \"ovnkube-node-xx8t5\" (UID: \"0568938d-c9e2-4e37-b768-cdeddaacf33e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.152382 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d5cb641-099e-4366-9d79-85d72a0e1a90-hostroot\") pod \"multus-wwvxj\" (UID: \"9d5cb641-099e-4366-9d79-85d72a0e1a90\") " pod="openshift-multus/multus-wwvxj" Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.152430 4227 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:54.154105 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.152471 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:54.652461015 +0000 UTC m=+1.889438893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:54.256314 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.256281 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:28:54.656748 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.656722 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:54.656884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.656755 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:54.656884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.656790 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:54.656884 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.656868 4227 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:54.657038 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.656913 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:55.656901961 +0000 UTC m=+2.893879813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:54.657038 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.656928 4227 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:54.657038 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.656871 4227 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:54.657038 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.656993 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:55.656979854 +0000 UTC m=+2.893957718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:54.657038 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:54.657010 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:55.657001055 +0000 UTC m=+2.893978908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : secret "metrics-daemon-secret" not found Nov 03 20:28:54.699774 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:54.699752 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:55.664454 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:55.664418 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:55.664462 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:55.664485 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:55.664599 4227 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:55.664603 4227 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:55.664654 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.66464222 +0000 UTC m=+4.901620085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:55.664678 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.664664708 +0000 UTC m=+4.901642560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:55.664603 4227 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:55.665012 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:55.664715 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.664704812 +0000 UTC m=+4.901682678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : secret "metrics-daemon-secret" not found Nov 03 20:28:55.881958 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:55.881933 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hc7pp" Nov 03 20:28:56.063957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:56.063874 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:28:56.064107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:56.064079 4227 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:28:57.674249 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:57.674204 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:57.674267 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:28:57.674290 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:57.674366 4227 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:57.674373 4227 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:57.674411 4227 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:57.674424 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:01.674411874 +0000 UTC m=+8.911389739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : secret "metrics-daemon-secret" not found Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:57.674437 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:01.674430928 +0000 UTC m=+8.911408780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:28:57.674871 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:28:57.674465 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:01.674452677 +0000 UTC m=+8.911430529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:00.857483 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:00.857448 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:29:01.693905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:01.693838 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:29:01.694073 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:01.693929 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:29:01.694073 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:01.693966 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:29:01.694073 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:01.694013 4227 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:01.694073 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:01.694072 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:09.694057982 +0000 UTC m=+16.931035837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:01.694242 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:01.694086 4227 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:29:01.694242 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:01.694140 4227 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:01.694242 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:01.694143 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:09.694130435 +0000 UTC m=+16.931108287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : secret "metrics-daemon-secret" not found Nov 03 20:29:01.694242 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:01.694193 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:09.69418127 +0000 UTC m=+16.931159125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:29:03.259806 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:03.259782 4227 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-71.us-west-2.compute.internal" event="NodeReady" Nov 03 20:29:05.957973 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:05.957940 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8mp2t" event={"ID":"44a82b10-e0ed-4945-8f9a-845487bbb556","Type":"ContainerStarted","Data":"f6579215f063d62fa2c0428d82adb9680abb796c48f35532ed7272c7ea4dd4f7"} Nov 03 20:29:06.080983 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:06.080912 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx8t5" Nov 03 20:29:09.756344 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:09.756307 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:29:09.756344 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:09.756347 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:29:09.756750 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:09.756368 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:29:09.756750 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:09.756453 4227 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:09.756750 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:09.756456 4227 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:09.756750 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:09.756498 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls podName:2964a2b0-45fb-4282-8cde-f2ef72de59c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:25.756487518 +0000 UTC m=+32.993465370 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls") pod "dns-default-9k29p" (UID: "2964a2b0-45fb-4282-8cde-f2ef72de59c5") : secret "dns-default-metrics-tls" not found Nov 03 20:29:09.756750 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:09.756455 4227 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:29:09.756750 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:09.756512 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls podName:ddd648a5-3a08-42c1-b8a3-e14ae414263c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:25.756506115 +0000 UTC m=+32.993483966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wcndc" (UID: "ddd648a5-3a08-42c1-b8a3-e14ae414263c") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:09.756750 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:09.756533 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs podName:e55a9e03-4858-42b2-a27a-a9533e404588 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:25.756517658 +0000 UTC m=+32.993495525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs") pod "network-metrics-daemon-xqvnn" (UID: "e55a9e03-4858-42b2-a27a-a9533e404588") : secret "metrics-daemon-secret" not found Nov 03 20:29:11.840961 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.840928 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6599b5d5cf-rx72s"] Nov 03 20:29:11.843737 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.843715 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869080 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869052 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-registry-tls\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869205 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869086 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-bound-sa-token\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869205 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869111 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqrl\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-kube-api-access-4wqrl\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869248 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7cacd8f-edc3-4a47-b558-2d06fdc78048-ca-trust-extracted\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869274 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7cacd8f-edc3-4a47-b558-2d06fdc78048-installation-pull-secrets\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869389 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869371 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7cacd8f-edc3-4a47-b558-2d06fdc78048-registry-certificates\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869436 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869424 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7cacd8f-edc3-4a47-b558-2d06fdc78048-trusted-ca\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.869469 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.869452 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7cacd8f-edc3-4a47-b558-2d06fdc78048-image-registry-private-configuration\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.893040 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.893017 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:29:11.939479 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.939450 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:29:11.939595 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.939549 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwh5m\"" Nov 03 20:29:11.939715 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.939698 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:29:11.970568 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970538 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7cacd8f-edc3-4a47-b558-2d06fdc78048-image-registry-private-configuration\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.970691 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970588 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-registry-tls\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.970691 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970614 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-bound-sa-token\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.970797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970726 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqrl\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-kube-api-access-4wqrl\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.970797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970755 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7cacd8f-edc3-4a47-b558-2d06fdc78048-ca-trust-extracted\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.970797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970773 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7cacd8f-edc3-4a47-b558-2d06fdc78048-installation-pull-secrets\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.970937 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970880 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7cacd8f-edc3-4a47-b558-2d06fdc78048-registry-certificates\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.970937 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.970927 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7cacd8f-edc3-4a47-b558-2d06fdc78048-trusted-ca\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.971175 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.971141 4227 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:29:11.971317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.971297 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7cacd8f-edc3-4a47-b558-2d06fdc78048-ca-trust-extracted\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.971791 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.971770 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7cacd8f-edc3-4a47-b558-2d06fdc78048-registry-certificates\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.974495 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.974478 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-registry-tls\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.974570 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.974476 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7cacd8f-edc3-4a47-b558-2d06fdc78048-installation-pull-secrets\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:11.974570 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:11.974480 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7cacd8f-edc3-4a47-b558-2d06fdc78048-image-registry-private-configuration\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:12.034148 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.033553 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:29:12.036306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.035732 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6599b5d5cf-rx72s"] Nov 03 20:29:12.036764 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.036740 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-bound-sa-token\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:12.036865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.036835 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqrl\" (UniqueName: \"kubernetes.io/projected/b7cacd8f-edc3-4a47-b558-2d06fdc78048-kube-api-access-4wqrl\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:12.041995 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.041969 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7cacd8f-edc3-4a47-b558-2d06fdc78048-trusted-ca\") pod \"image-registry-6599b5d5cf-rx72s\" (UID: \"b7cacd8f-edc3-4a47-b558-2d06fdc78048\") " pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:12.154210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.154185 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:12.567946 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.567877 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6599b5d5cf-rx72s"] Nov 03 20:29:12.575204 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:12.575179 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cacd8f_edc3_4a47_b558_2d06fdc78048.slice/crio-be34f0cefb85ea5f92aa44c257e030f8c585018760fbb1b4524c0e882fd3a163 WatchSource:0}: Error finding container be34f0cefb85ea5f92aa44c257e030f8c585018760fbb1b4524c0e882fd3a163: Status 404 returned error can't find the container with id be34f0cefb85ea5f92aa44c257e030f8c585018760fbb1b4524c0e882fd3a163 Nov 03 20:29:12.977575 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.977537 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" event={"ID":"b7cacd8f-edc3-4a47-b558-2d06fdc78048","Type":"ContainerStarted","Data":"93b663ccabce792a30b467864dfbbb626759cb8c1dbde10e392d3219f22aa977"} Nov 03 20:29:12.977950 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.977593 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" event={"ID":"b7cacd8f-edc3-4a47-b558-2d06fdc78048","Type":"ContainerStarted","Data":"be34f0cefb85ea5f92aa44c257e030f8c585018760fbb1b4524c0e882fd3a163"} Nov 03 20:29:12.977950 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:12.977682 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:13.087337 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:13.087278 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" podStartSLOduration=2.087259839 podStartE2EDuration="2.087259839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:13.086569239 +0000 UTC m=+20.323547114" watchObservedRunningTime="2026-11-03 20:29:13.087259839 +0000 UTC m=+20.324237712" Nov 03 20:29:20.859976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:20.859948 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7v5wf" Nov 03 20:29:23.724580 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.724551 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gwrtb"] Nov 03 20:29:23.754686 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.754664 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:23.781817 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.781792 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gwrtb"] Nov 03 20:29:23.803479 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.803453 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:29:23.803479 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.803456 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:29:23.803761 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.803749 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gxdgg\"" Nov 03 20:29:23.852363 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.852342 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2c0eed1-2041-47fc-81f5-e75899b0cbc9-tmp\") pod \"downloads-6dd789c8b7-gwrtb\" (UID: \"b2c0eed1-2041-47fc-81f5-e75899b0cbc9\") " pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:23.852461 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.852391 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2djxb\" (UniqueName: \"kubernetes.io/projected/b2c0eed1-2041-47fc-81f5-e75899b0cbc9-kube-api-access-2djxb\") pod \"downloads-6dd789c8b7-gwrtb\" (UID: \"b2c0eed1-2041-47fc-81f5-e75899b0cbc9\") " pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:23.953472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.953447 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2c0eed1-2041-47fc-81f5-e75899b0cbc9-tmp\") pod \"downloads-6dd789c8b7-gwrtb\" (UID: \"b2c0eed1-2041-47fc-81f5-e75899b0cbc9\") " pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:23.953551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.953500 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2djxb\" (UniqueName: \"kubernetes.io/projected/b2c0eed1-2041-47fc-81f5-e75899b0cbc9-kube-api-access-2djxb\") pod \"downloads-6dd789c8b7-gwrtb\" (UID: \"b2c0eed1-2041-47fc-81f5-e75899b0cbc9\") " pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:23.953909 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:23.953894 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2c0eed1-2041-47fc-81f5-e75899b0cbc9-tmp\") pod \"downloads-6dd789c8b7-gwrtb\" (UID: \"b2c0eed1-2041-47fc-81f5-e75899b0cbc9\") " pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:24.006885 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:24.006846 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2djxb\" (UniqueName: \"kubernetes.io/projected/b2c0eed1-2041-47fc-81f5-e75899b0cbc9-kube-api-access-2djxb\") pod \"downloads-6dd789c8b7-gwrtb\" (UID: \"b2c0eed1-2041-47fc-81f5-e75899b0cbc9\") " pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:24.085271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:24.085237 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:24.572548 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:24.572516 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gwrtb"] Nov 03 20:29:24.622611 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:24.622584 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c0eed1_2041_47fc_81f5_e75899b0cbc9.slice/crio-f57ba0848460368eb8861c7cd84bcc59d474df2b6a224bdee6eba7b2fef40a93 WatchSource:0}: Error finding container f57ba0848460368eb8861c7cd84bcc59d474df2b6a224bdee6eba7b2fef40a93: Status 404 returned error can't find the container with id f57ba0848460368eb8861c7cd84bcc59d474df2b6a224bdee6eba7b2fef40a93 Nov 03 20:29:25.017303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:25.017260 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gwrtb" event={"ID":"b2c0eed1-2041-47fc-81f5-e75899b0cbc9","Type":"ContainerStarted","Data":"f57ba0848460368eb8861c7cd84bcc59d474df2b6a224bdee6eba7b2fef40a93"} Nov 03 20:29:25.767804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:25.767765 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:29:25.767991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:25.767817 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:29:25.767991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:25.767849 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:29:25.770704 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:25.770671 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2964a2b0-45fb-4282-8cde-f2ef72de59c5-metrics-tls\") pod \"dns-default-9k29p\" (UID: \"2964a2b0-45fb-4282-8cde-f2ef72de59c5\") " pod="openshift-dns/dns-default-9k29p" Nov 03 20:29:25.770799 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:25.770670 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55a9e03-4858-42b2-a27a-a9533e404588-metrics-certs\") pod \"network-metrics-daemon-xqvnn\" (UID: \"e55a9e03-4858-42b2-a27a-a9533e404588\") " pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:29:25.771406 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:25.771383 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd648a5-3a08-42c1-b8a3-e14ae414263c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcndc\" (UID: \"ddd648a5-3a08-42c1-b8a3-e14ae414263c\") " pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:29:26.052726 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:26.052658 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqvnn" Nov 03 20:29:26.053766 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:26.053748 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wcndc" Nov 03 20:29:26.054130 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:26.054110 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9k29p" Nov 03 20:29:26.435655 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:26.435625 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9k29p"] Nov 03 20:29:26.475982 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:26.475955 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2964a2b0_45fb_4282_8cde_f2ef72de59c5.slice/crio-f034ae3815959d067d5e4b814e7056c632b41c8dc3c02e9fb11008d846ae21fb WatchSource:0}: Error finding container f034ae3815959d067d5e4b814e7056c632b41c8dc3c02e9fb11008d846ae21fb: Status 404 returned error can't find the container with id f034ae3815959d067d5e4b814e7056c632b41c8dc3c02e9fb11008d846ae21fb Nov 03 20:29:26.481963 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:26.481940 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xqvnn"] Nov 03 20:29:26.482454 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:26.482438 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wcndc"] Nov 03 20:29:26.486370 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:26.486344 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55a9e03_4858_42b2_a27a_a9533e404588.slice/crio-41695b94cadb6a9216df0ba7ddeea9fefb5b646637bbe10b13569ba9be5a6a8f WatchSource:0}: Error finding container 41695b94cadb6a9216df0ba7ddeea9fefb5b646637bbe10b13569ba9be5a6a8f: Status 404 returned error can't find the container with id 41695b94cadb6a9216df0ba7ddeea9fefb5b646637bbe10b13569ba9be5a6a8f Nov 03 20:29:26.495948 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:26.495923 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd648a5_3a08_42c1_b8a3_e14ae414263c.slice/crio-5d18a911d156c4b57067a76e6c57611d1fa237fdec41617421384312a616f542 WatchSource:0}: Error finding container 5d18a911d156c4b57067a76e6c57611d1fa237fdec41617421384312a616f542: Status 404 returned error can't find the container with id 5d18a911d156c4b57067a76e6c57611d1fa237fdec41617421384312a616f542 Nov 03 20:29:27.020249 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:27.020222 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wjxm_aa839ba6-7b29-41c8-bd50-a7cd5b1ac139/dns-node-resolver/0.log" Nov 03 20:29:27.022809 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:27.022785 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9k29p" event={"ID":"2964a2b0-45fb-4282-8cde-f2ef72de59c5","Type":"ContainerStarted","Data":"f034ae3815959d067d5e4b814e7056c632b41c8dc3c02e9fb11008d846ae21fb"} Nov 03 20:29:27.024140 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:27.024117 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcndc" event={"ID":"ddd648a5-3a08-42c1-b8a3-e14ae414263c","Type":"ContainerStarted","Data":"59509ef90d7b20e301517754d2aa0587819da67bc07af70c9578514f109a2d21"} Nov 03 20:29:27.024246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:27.024185 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcndc" event={"ID":"ddd648a5-3a08-42c1-b8a3-e14ae414263c","Type":"ContainerStarted","Data":"5d18a911d156c4b57067a76e6c57611d1fa237fdec41617421384312a616f542"} Nov 03 20:29:27.025118 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:27.025097 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xqvnn" event={"ID":"e55a9e03-4858-42b2-a27a-a9533e404588","Type":"ContainerStarted","Data":"41695b94cadb6a9216df0ba7ddeea9fefb5b646637bbe10b13569ba9be5a6a8f"} Nov 03 20:29:27.620511 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:27.620475 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6599b5d5cf-rx72s_b7cacd8f-edc3-4a47-b558-2d06fdc78048/registry/0.log" Nov 03 20:29:28.029796 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:28.029765 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcndc" event={"ID":"ddd648a5-3a08-42c1-b8a3-e14ae414263c","Type":"ContainerStarted","Data":"b24a5d426bc8e72e3d014f733eefe41aa9c032b7f0a80764a2d1d097c3036ff9"} Nov 03 20:29:28.220380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:28.220312 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dg8z2_11a94da7-4353-4baf-9cd9-e585ff91ac82/node-ca/0.log" Nov 03 20:29:28.916677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:28.916645 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8"] Nov 03 20:29:28.919058 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:28.919029 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" Nov 03 20:29:28.990420 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:28.990307 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/71ff80c2-3f12-4450-9b85-32338106f953-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-spxv8\" (UID: \"71ff80c2-3f12-4450-9b85-32338106f953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" Nov 03 20:29:29.014437 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.014406 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tpssf\"" Nov 03 20:29:29.014591 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.014515 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 20:29:29.015019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.014993 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8"] Nov 03 20:29:29.035114 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.035082 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xqvnn" event={"ID":"e55a9e03-4858-42b2-a27a-a9533e404588","Type":"ContainerStarted","Data":"6f550b37b23d8f1bb24497214d4390f61db358b0ba552459fdc9b60175beb372"} Nov 03 20:29:29.035226 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.035133 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xqvnn" event={"ID":"e55a9e03-4858-42b2-a27a-a9533e404588","Type":"ContainerStarted","Data":"145933a02bcab77bae9db34add33c510274912868939c6c354bc1ffa09427695"} Nov 03 20:29:29.091417 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.091394 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/71ff80c2-3f12-4450-9b85-32338106f953-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-spxv8\" (UID: \"71ff80c2-3f12-4450-9b85-32338106f953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" Nov 03 20:29:29.094638 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.094608 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/71ff80c2-3f12-4450-9b85-32338106f953-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-spxv8\" (UID: \"71ff80c2-3f12-4450-9b85-32338106f953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" Nov 03 20:29:29.220764 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.220730 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lrdnf_40a6b42d-1468-4f9b-85f3-4243a25e01ac/serve-healthcheck-canary/0.log" Nov 03 20:29:29.233455 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.233428 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" Nov 03 20:29:29.613682 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:29.613597 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8"] Nov 03 20:29:29.623039 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:29.623007 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ff80c2_3f12_4450_9b85_32338106f953.slice/crio-b73bcd330c870352d5fa2231c03a3ff457e3898f603346c321d7a8c7b9a2b6e0 WatchSource:0}: Error finding container b73bcd330c870352d5fa2231c03a3ff457e3898f603346c321d7a8c7b9a2b6e0: Status 404 returned error can't find the container with id b73bcd330c870352d5fa2231c03a3ff457e3898f603346c321d7a8c7b9a2b6e0 Nov 03 20:29:30.040879 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:30.040845 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9k29p" event={"ID":"2964a2b0-45fb-4282-8cde-f2ef72de59c5","Type":"ContainerStarted","Data":"b262f319b9d483e9e59fa99ad31dbc2bc95acc17f8243197aa7b4e62bc5d968e"} Nov 03 20:29:30.041294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:30.040904 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9k29p" event={"ID":"2964a2b0-45fb-4282-8cde-f2ef72de59c5","Type":"ContainerStarted","Data":"9673852a99223f49456fa8cf3e6ce5131b299a14b7a4a28373290567ec9bf18a"} Nov 03 20:29:30.041629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:30.041613 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9k29p" Nov 03 20:29:30.043946 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:30.043911 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcndc" event={"ID":"ddd648a5-3a08-42c1-b8a3-e14ae414263c","Type":"ContainerStarted","Data":"1bdd7dc9295fb003101025b1d6f761e551bce76f04c51316559d5335970f4df0"} Nov 03 20:29:30.045110 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:30.045090 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" event={"ID":"71ff80c2-3f12-4450-9b85-32338106f953","Type":"ContainerStarted","Data":"b73bcd330c870352d5fa2231c03a3ff457e3898f603346c321d7a8c7b9a2b6e0"} Nov 03 20:29:32.055611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:32.055573 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" event={"ID":"71ff80c2-3f12-4450-9b85-32338106f953","Type":"ContainerStarted","Data":"4578d568a46746666f203b57c3858cf85215e9ab5e751d21cf7792c6bc9a27cc"} Nov 03 20:29:32.056050 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:32.055770 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" Nov 03 20:29:32.061431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:32.061406 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" Nov 03 20:29:32.158792 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:32.158759 4227 patch_prober.go:29] interesting pod/image-registry-6599b5d5cf-rx72s container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:29:32.158913 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:32.158808 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" podUID="b7cacd8f-edc3-4a47-b558-2d06fdc78048" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:32.160584 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:32.160320 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-spxv8" podStartSLOduration=2.516215816 podStartE2EDuration="4.160307138s" totalImagesPullingTime="1.644091322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:32.159567164 +0000 UTC m=+39.396545039" watchObservedRunningTime="2026-11-03 20:29:32.160307138 +0000 UTC m=+39.397285012" Nov 03 20:29:33.986610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:33.986579 4227 patch_prober.go:29] interesting pod/image-registry-6599b5d5cf-rx72s container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:29:33.986966 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:33.986622 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" podUID="b7cacd8f-edc3-4a47-b558-2d06fdc78048" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:34.351642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.351517 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57df498d94-mcrcw"] Nov 03 20:29:34.353776 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.353712 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.402878 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.402852 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 20:29:34.428641 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.428617 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-serving-cert\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.428641 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.428645 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-oauth-config\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.428793 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.428665 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1437e0e-5461-43c3-a7ed-712b006033da-tmp\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.428793 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.428689 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbftn\" (UniqueName: \"kubernetes.io/projected/c1437e0e-5461-43c3-a7ed-712b006033da-kube-api-access-rbftn\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.428911 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.428882 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-console-config\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.428970 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.428934 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-service-ca\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.429039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.429020 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-oauth-serving-cert\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.449122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.449096 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 20:29:34.449265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.449172 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 20:29:34.449265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.449233 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 20:29:34.449365 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.449356 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 20:29:34.449413 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.449369 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p55vv\"" Nov 03 20:29:34.450095 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.450080 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57df498d94-mcrcw"] Nov 03 20:29:34.529832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.529785 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-serving-cert\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.529832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.529831 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-oauth-config\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.529854 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1437e0e-5461-43c3-a7ed-712b006033da-tmp\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.529882 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbftn\" (UniqueName: \"kubernetes.io/projected/c1437e0e-5461-43c3-a7ed-712b006033da-kube-api-access-rbftn\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.529916 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-console-config\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.529948 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-service-ca\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.530147 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-oauth-serving-cert\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.530426 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1437e0e-5461-43c3-a7ed-712b006033da-tmp\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530746 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.530725 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-console-config\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530842 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.530825 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-oauth-serving-cert\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.530875 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.530844 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-service-ca\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.532336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.532315 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-serving-cert\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.532432 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.532319 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-oauth-config\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.582707 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.582685 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbftn\" (UniqueName: \"kubernetes.io/projected/c1437e0e-5461-43c3-a7ed-712b006033da-kube-api-access-rbftn\") pod \"console-57df498d94-mcrcw\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:34.663814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:34.663792 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:35.032922 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:35.032841 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57df498d94-mcrcw"] Nov 03 20:29:35.040930 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:35.040888 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1437e0e_5461_43c3_a7ed_712b006033da.slice/crio-6cb58162da746aea527a5f8fe152232b0fe12f95ce6983ba699ed7ca4fcaa487 WatchSource:0}: Error finding container 6cb58162da746aea527a5f8fe152232b0fe12f95ce6983ba699ed7ca4fcaa487: Status 404 returned error can't find the container with id 6cb58162da746aea527a5f8fe152232b0fe12f95ce6983ba699ed7ca4fcaa487 Nov 03 20:29:35.065452 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:35.065428 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57df498d94-mcrcw" event={"ID":"c1437e0e-5461-43c3-a7ed-712b006033da","Type":"ContainerStarted","Data":"6cb58162da746aea527a5f8fe152232b0fe12f95ce6983ba699ed7ca4fcaa487"} Nov 03 20:29:39.079298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:39.079267 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57df498d94-mcrcw" event={"ID":"c1437e0e-5461-43c3-a7ed-712b006033da","Type":"ContainerStarted","Data":"375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e"} Nov 03 20:29:39.187292 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:39.187244 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57df498d94-mcrcw" podStartSLOduration=1.9172034999999998 podStartE2EDuration="5.187228428s" totalImagesPullingTime="3.270024928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:39.186618164 +0000 UTC m=+46.423596040" watchObservedRunningTime="2026-11-03 20:29:39.187228428 +0000 UTC m=+46.424206303" Nov 03 20:29:41.057049 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:41.057020 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9k29p" Nov 03 20:29:42.158933 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:42.158895 4227 patch_prober.go:29] interesting pod/image-registry-6599b5d5cf-rx72s container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:29:42.159588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:42.158951 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" podUID="b7cacd8f-edc3-4a47-b558-2d06fdc78048" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:43.986252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:43.986221 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6599b5d5cf-rx72s" Nov 03 20:29:44.664995 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:44.664958 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:44.665215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:44.665031 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:44.670647 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:44.670624 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:45.110332 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.110264 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:29:45.266435 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.266400 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jq2dt"] Nov 03 20:29:45.269915 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.269878 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.324227 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.324053 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:29:45.324227 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.324085 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2c9vt\"" Nov 03 20:29:45.324227 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.324199 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:29:45.397362 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.397280 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:29:45.397541 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.397397 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:29:45.397541 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.397468 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:29:45.397661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.397594 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:29:45.417049 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417012 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417242 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417054 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417242 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417133 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-root\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417242 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417172 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-sys\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417444 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417251 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-tls\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417444 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417284 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e664735-47ec-43df-8b7e-32b12315bf4e-metrics-client-ca\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417444 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417304 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmr7\" (UniqueName: \"kubernetes.io/projected/4e664735-47ec-43df-8b7e-32b12315bf4e-kube-api-access-gkmr7\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417444 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417341 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-textfile\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.417444 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.417367 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-wtmp\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518008 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.517971 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-root\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518022 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-sys\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518050 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-tls\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518078 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e664735-47ec-43df-8b7e-32b12315bf4e-metrics-client-ca\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518099 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-root\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518187 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-sys\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518108 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmr7\" (UniqueName: \"kubernetes.io/projected/4e664735-47ec-43df-8b7e-32b12315bf4e-kube-api-access-gkmr7\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518485 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-textfile\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518519 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-wtmp\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518623 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518571 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.518623 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.518611 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.519069 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.519046 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e664735-47ec-43df-8b7e-32b12315bf4e-metrics-client-ca\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.519147 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.519126 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-wtmp\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.519343 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.519323 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-textfile\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.521651 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.521576 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.522063 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.522022 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-tls\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.526284 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.526230 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e664735-47ec-43df-8b7e-32b12315bf4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.574237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.574202 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmr7\" (UniqueName: \"kubernetes.io/projected/4e664735-47ec-43df-8b7e-32b12315bf4e-kube-api-access-gkmr7\") pod \"node-exporter-jq2dt\" (UID: \"4e664735-47ec-43df-8b7e-32b12315bf4e\") " pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.588839 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:45.588813 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jq2dt" Nov 03 20:29:45.600846 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:45.600804 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e664735_47ec_43df_8b7e_32b12315bf4e.slice/crio-cbda3d8e202438e639d7c0acf7eedc0594d284439bbd0ecebe1e923814af8d45 WatchSource:0}: Error finding container cbda3d8e202438e639d7c0acf7eedc0594d284439bbd0ecebe1e923814af8d45: Status 404 returned error can't find the container with id cbda3d8e202438e639d7c0acf7eedc0594d284439bbd0ecebe1e923814af8d45 Nov 03 20:29:46.102245 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:46.102203 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jq2dt" event={"ID":"4e664735-47ec-43df-8b7e-32b12315bf4e","Type":"ContainerStarted","Data":"cbda3d8e202438e639d7c0acf7eedc0594d284439bbd0ecebe1e923814af8d45"} Nov 03 20:29:47.344305 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.344275 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:29:47.347016 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.346991 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.398018 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.397997 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 20:29:47.398110 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.398019 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 20:29:47.399395 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.398781 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 20:29:47.399395 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.398896 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 20:29:47.400142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.400119 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:29:47.435945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.435924 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436033 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.435960 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436033 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.435977 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436100 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436030 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436100 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436059 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-web-config\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436115 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-out\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436138 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436257 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436190 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436257 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436231 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-volume\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436315 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436289 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jrs\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-kube-api-access-w9jrs\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436315 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436308 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436412 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436347 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.436412 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.436368 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.441607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.441590 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 20:29:47.492455 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.492395 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 20:29:47.500031 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.500009 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 20:29:47.500031 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.500021 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l6mwl\"" Nov 03 20:29:47.500149 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.500060 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 20:29:47.500149 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.500081 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 20:29:47.537431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537403 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537591 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537458 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537591 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537488 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537591 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537517 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537623 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537660 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537694 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537721 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-web-config\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537749 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-out\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537772 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537795 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537830 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-volume\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.537942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.537872 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jrs\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-kube-api-access-w9jrs\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.540440 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.538515 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.540440 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.539224 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.540440 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.539441 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.541783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.541610 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.541783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.541636 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.542520 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.542490 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.543245 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.543218 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-web-config\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.543641 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.543611 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.543853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.543831 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.544292 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.544269 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-out\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.544475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.544457 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-volume\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.547032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.547009 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.591041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.591013 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jrs\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-kube-api-access-w9jrs\") pod \"alertmanager-main-0\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:47.731858 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:47.731821 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:29:48.110875 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:48.110803 4227 generic.go:356] "Generic (PLEG): container finished" podID="4e664735-47ec-43df-8b7e-32b12315bf4e" containerID="4c61116c076d7e62993431103b9140ee956d6b51a59b62bfb5c6fb01ec99c951" exitCode=0 Nov 03 20:29:48.111032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:48.110876 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jq2dt" event={"ID":"4e664735-47ec-43df-8b7e-32b12315bf4e","Type":"ContainerDied","Data":"4c61116c076d7e62993431103b9140ee956d6b51a59b62bfb5c6fb01ec99c951"} Nov 03 20:29:48.218315 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:48.218290 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:29:48.224688 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:48.224663 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85a35ae_3cc8_47fa_b85a_4d548692c113.slice/crio-2253ec496977edac301b128c5c01dcd50b6ef12b56a62909b8a02bb0e39c5c80 WatchSource:0}: Error finding container 2253ec496977edac301b128c5c01dcd50b6ef12b56a62909b8a02bb0e39c5c80: Status 404 returned error can't find the container with id 2253ec496977edac301b128c5c01dcd50b6ef12b56a62909b8a02bb0e39c5c80 Nov 03 20:29:49.116373 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:49.116335 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jq2dt" event={"ID":"4e664735-47ec-43df-8b7e-32b12315bf4e","Type":"ContainerStarted","Data":"6f0a6e2f15938ca8fd159723cfa9d3d4cfd5a7da64eb3e202c51343df58728f8"} Nov 03 20:29:49.116872 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:49.116391 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jq2dt" event={"ID":"4e664735-47ec-43df-8b7e-32b12315bf4e","Type":"ContainerStarted","Data":"91e96ed668e93b0f65687cb44afc14dea17989971dab905beaa5acbe0604d360"} Nov 03 20:29:49.117702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:49.117671 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerStarted","Data":"2253ec496977edac301b128c5c01dcd50b6ef12b56a62909b8a02bb0e39c5c80"} Nov 03 20:29:49.225829 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:49.225776 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jq2dt" podStartSLOduration=2.558733264 podStartE2EDuration="4.225756983s" totalImagesPullingTime="1.59988901s" totalInitContainerRuntime="67.134709ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:49.224536354 +0000 UTC m=+56.461514230" watchObservedRunningTime="2026-11-03 20:29:49.225756983 +0000 UTC m=+56.462734859" Nov 03 20:29:50.123281 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:50.123192 4227 generic.go:356] "Generic (PLEG): container finished" podID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerID="01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5" exitCode=0 Nov 03 20:29:50.123740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:50.123272 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5"} Nov 03 20:29:50.980749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:50.980710 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k"] Nov 03 20:29:50.983569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:50.983544 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.034084 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.034053 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 20:29:51.034582 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.034064 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 20:29:51.034712 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.034612 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 20:29:51.034933 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.034902 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 20:29:51.035083 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.035060 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7gkkt\"" Nov 03 20:29:51.069378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069346 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-secret-telemeter-client\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.069378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069381 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-metrics-client-ca\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.069562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069414 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-telemeter-client-tls\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.069609 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069567 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-serving-certs-ca-bundle\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.069609 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069595 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.069692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069681 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-federate-client-tls\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.069722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069699 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.069773 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.069721 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcpd\" (UniqueName: \"kubernetes.io/projected/08f9c3db-d42c-4367-891c-eca082d6e806-kube-api-access-kzcpd\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.080275 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.080247 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 20:29:51.127355 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.127304 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 20:29:51.129245 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.129216 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k"] Nov 03 20:29:51.171083 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171047 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-federate-client-tls\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.171083 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171089 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.171325 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171226 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcpd\" (UniqueName: \"kubernetes.io/projected/08f9c3db-d42c-4367-891c-eca082d6e806-kube-api-access-kzcpd\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.171325 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171266 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-secret-telemeter-client\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.171325 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171295 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-metrics-client-ca\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.171473 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171336 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-telemeter-client-tls\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.171473 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171408 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-serving-certs-ca-bundle\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.171473 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.171433 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.172041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.172007 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-metrics-client-ca\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.172178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.172102 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.172245 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.172207 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9c3db-d42c-4367-891c-eca082d6e806-serving-certs-ca-bundle\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.174279 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.174258 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-federate-client-tls\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.174405 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.174386 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-telemeter-client-tls\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.174485 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.174387 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-secret-telemeter-client\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.174485 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.174453 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08f9c3db-d42c-4367-891c-eca082d6e806-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.224737 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.224706 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcpd\" (UniqueName: \"kubernetes.io/projected/08f9c3db-d42c-4367-891c-eca082d6e806-kube-api-access-kzcpd\") pod \"telemeter-client-6bcf87d6c9-ck62k\" (UID: \"08f9c3db-d42c-4367-891c-eca082d6e806\") " pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.299597 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.299512 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" Nov 03 20:29:51.728741 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:51.728705 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k"] Nov 03 20:29:51.738065 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:51.738031 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f9c3db_d42c_4367_891c_eca082d6e806.slice/crio-7d526864ddfd5f65045afa7a745fab4fb53de8763ee53f841f02cdf73009494a WatchSource:0}: Error finding container 7d526864ddfd5f65045afa7a745fab4fb53de8763ee53f841f02cdf73009494a: Status 404 returned error can't find the container with id 7d526864ddfd5f65045afa7a745fab4fb53de8763ee53f841f02cdf73009494a Nov 03 20:29:52.132550 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:52.132468 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" event={"ID":"08f9c3db-d42c-4367-891c-eca082d6e806","Type":"ContainerStarted","Data":"7d526864ddfd5f65045afa7a745fab4fb53de8763ee53f841f02cdf73009494a"} Nov 03 20:29:53.140850 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:53.140809 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerStarted","Data":"c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a"} Nov 03 20:29:53.141310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:53.140925 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerStarted","Data":"87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049"} Nov 03 20:29:53.141310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:53.140962 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerStarted","Data":"8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb"} Nov 03 20:29:53.141310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:53.141003 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerStarted","Data":"a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771"} Nov 03 20:29:53.141310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:53.141041 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerStarted","Data":"bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7"} Nov 03 20:29:54.149386 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:54.149345 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerStarted","Data":"dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734"} Nov 03 20:29:54.404696 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:54.404644 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.2719024 podStartE2EDuration="7.404624847s" totalImagesPullingTime="5.118921215s" totalInitContainerRuntime="13.801232ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:54.357601131 +0000 UTC m=+61.594579028" watchObservedRunningTime="2026-11-03 20:29:54.404624847 +0000 UTC m=+61.641602722" Nov 03 20:29:55.155312 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.155274 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" event={"ID":"08f9c3db-d42c-4367-891c-eca082d6e806","Type":"ContainerStarted","Data":"91fdbae08028dae1f5cd4518b0679c39df8fd6b98a0af6e617632dc9ffae7c8f"} Nov 03 20:29:55.155677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.155344 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" event={"ID":"08f9c3db-d42c-4367-891c-eca082d6e806","Type":"ContainerStarted","Data":"6eb39c991a5d2818c56b06a8b7b59724f3b46b2dcb59c2d90ed3e5cd29b79167"} Nov 03 20:29:55.155677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.155364 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" event={"ID":"08f9c3db-d42c-4367-891c-eca082d6e806","Type":"ContainerStarted","Data":"8e3931a0f1ef1f1dc930113925311923a169657a73c94ea53def6d7e5c048f42"} Nov 03 20:29:55.156836 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.156810 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gwrtb" event={"ID":"b2c0eed1-2041-47fc-81f5-e75899b0cbc9","Type":"ContainerStarted","Data":"e3b05e6bdb8bd181e4f54d4b698bcd5402e1429488d719d2cc7f5c49295693d2"} Nov 03 20:29:55.157213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.157193 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:55.158414 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.158396 4227 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-gwrtb container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body= Nov 03 20:29:55.158474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.158434 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-gwrtb" podUID="b2c0eed1-2041-47fc-81f5-e75899b0cbc9" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" Nov 03 20:29:55.265078 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.264532 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6bcf87d6c9-ck62k" podStartSLOduration=2.809793725 podStartE2EDuration="5.264516283s" totalImagesPullingTime="2.454722558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:55.263970096 +0000 UTC m=+62.500947996" watchObservedRunningTime="2026-11-03 20:29:55.264516283 +0000 UTC m=+62.501494157" Nov 03 20:29:55.378757 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.378638 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-gwrtb" podStartSLOduration=1.924424715 podStartE2EDuration="32.378620791s" totalImagesPullingTime="30.454196076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:55.378488726 +0000 UTC m=+62.615466625" watchObservedRunningTime="2026-11-03 20:29:55.378620791 +0000 UTC m=+62.615598669" Nov 03 20:29:55.575223 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:55.575183 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57df498d94-mcrcw"] Nov 03 20:29:56.090885 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:29:56.090839 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:29:56.177779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:56.177746 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gwrtb" Nov 03 20:29:57.651831 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.651795 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f44845446-btth6"] Nov 03 20:29:57.679465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.679434 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.830691 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.830660 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-service-ca\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.830874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.830699 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-tmp\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.830874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.830727 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-trusted-ca-bundle\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.830874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.830795 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fgk7\" (UniqueName: \"kubernetes.io/projected/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-kube-api-access-9fgk7\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.831017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.830910 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-serving-cert\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.831017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.830944 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-oauth-config\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.831017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.830972 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-oauth-serving-cert\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.831112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.831027 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-config\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.869137 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.869108 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 20:29:57.870139 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.870115 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f44845446-btth6"] Nov 03 20:29:57.932098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932026 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-service-ca\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932077 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-tmp\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932142 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-trusted-ca-bundle\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932204 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgk7\" (UniqueName: \"kubernetes.io/projected/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-kube-api-access-9fgk7\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932248 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-serving-cert\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932270 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-oauth-config\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932291 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-oauth-serving-cert\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932571 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932319 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-config\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.932571 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.932490 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-tmp\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.933330 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.933267 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-service-ca\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.933467 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.933347 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-oauth-serving-cert\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.933534 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.933516 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-config\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.934286 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.934262 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-trusted-ca-bundle\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.935356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.935334 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-oauth-config\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.935516 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.935495 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-serving-cert\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:57.985753 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:57.985718 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgk7\" (UniqueName: \"kubernetes.io/projected/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-kube-api-access-9fgk7\") pod \"console-6f44845446-btth6\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:58.006710 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:58.006682 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:29:58.405560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:58.405514 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f44845446-btth6"] Nov 03 20:29:58.447687 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:29:58.447659 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08ef4c9_2973_4c41_be8d_96a8b6ec423d.slice/crio-0bd2a0e85b494ac6302a648cccfee97916c3f8470cf008568caa79a59bc51bbe WatchSource:0}: Error finding container 0bd2a0e85b494ac6302a648cccfee97916c3f8470cf008568caa79a59bc51bbe: Status 404 returned error can't find the container with id 0bd2a0e85b494ac6302a648cccfee97916c3f8470cf008568caa79a59bc51bbe Nov 03 20:29:59.174629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:59.174585 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44845446-btth6" event={"ID":"c08ef4c9-2973-4c41-be8d-96a8b6ec423d","Type":"ContainerStarted","Data":"110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642"} Nov 03 20:29:59.175200 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:59.174653 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44845446-btth6" event={"ID":"c08ef4c9-2973-4c41-be8d-96a8b6ec423d","Type":"ContainerStarted","Data":"0bd2a0e85b494ac6302a648cccfee97916c3f8470cf008568caa79a59bc51bbe"} Nov 03 20:29:59.283042 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:29:59.282979 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f44845446-btth6" podStartSLOduration=2.2829661039999998 podStartE2EDuration="2.282966104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:59.281992846 +0000 UTC m=+66.518970721" watchObservedRunningTime="2026-11-03 20:29:59.282966104 +0000 UTC m=+66.519943978" Nov 03 20:30:08.007055 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:08.007011 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:30:08.007563 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:08.007083 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:30:08.011772 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:08.011750 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:30:08.211902 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:08.211869 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:30:20.601112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.601074 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57df498d94-mcrcw" podUID="c1437e0e-5461-43c3-a7ed-712b006033da" containerName="console" containerID="cri-o://375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e" gracePeriod=15 Nov 03 20:30:20.839769 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.839746 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:30:20.906142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906119 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1437e0e-5461-43c3-a7ed-712b006033da-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1437e0e-5461-43c3-a7ed-712b006033da-tmp\") pod \"c1437e0e-5461-43c3-a7ed-712b006033da\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " Nov 03 20:30:20.906290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906152 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-console-config\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-console-config\") pod \"c1437e0e-5461-43c3-a7ed-712b006033da\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " Nov 03 20:30:20.906290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906191 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-oauth-config\") pod \"c1437e0e-5461-43c3-a7ed-712b006033da\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " Nov 03 20:30:20.906290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906211 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-service-ca\") pod \"c1437e0e-5461-43c3-a7ed-712b006033da\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " Nov 03 20:30:20.906290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906227 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-oauth-serving-cert\") pod \"c1437e0e-5461-43c3-a7ed-712b006033da\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " Nov 03 20:30:20.906290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906276 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1437e0e-5461-43c3-a7ed-712b006033da-kube-api-access-rbftn\" (UniqueName: \"kubernetes.io/projected/c1437e0e-5461-43c3-a7ed-712b006033da-kube-api-access-rbftn\") pod \"c1437e0e-5461-43c3-a7ed-712b006033da\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " Nov 03 20:30:20.906545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906336 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-serving-cert\") pod \"c1437e0e-5461-43c3-a7ed-712b006033da\" (UID: \"c1437e0e-5461-43c3-a7ed-712b006033da\") " Nov 03 20:30:20.906545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906372 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1437e0e-5461-43c3-a7ed-712b006033da-tmp" pod "c1437e0e-5461-43c3-a7ed-712b006033da" (UID: "c1437e0e-5461-43c3-a7ed-712b006033da"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:20.906545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906534 4227 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1437e0e-5461-43c3-a7ed-712b006033da-tmp\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.906702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906615 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-console-config" pod "c1437e0e-5461-43c3-a7ed-712b006033da" (UID: "c1437e0e-5461-43c3-a7ed-712b006033da"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.906702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906631 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-service-ca" pod "c1437e0e-5461-43c3-a7ed-712b006033da" (UID: "c1437e0e-5461-43c3-a7ed-712b006033da"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.906702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.906653 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-oauth-serving-cert" pod "c1437e0e-5461-43c3-a7ed-712b006033da" (UID: "c1437e0e-5461-43c3-a7ed-712b006033da"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.908385 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.908358 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-serving-cert" pod "c1437e0e-5461-43c3-a7ed-712b006033da" (UID: "c1437e0e-5461-43c3-a7ed-712b006033da"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:20.908463 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.908415 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1437e0e-5461-43c3-a7ed-712b006033da-kube-api-access-rbftn" pod "c1437e0e-5461-43c3-a7ed-712b006033da" (UID: "c1437e0e-5461-43c3-a7ed-712b006033da"). InnerVolumeSpecName "kube-api-access-rbftn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.908502 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:20.908476 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-oauth-config" pod "c1437e0e-5461-43c3-a7ed-712b006033da" (UID: "c1437e0e-5461-43c3-a7ed-712b006033da"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:21.006865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.006836 4227 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-console-config\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.006865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.006861 4227 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-oauth-config\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.006865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.006871 4227 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.007048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.006880 4227 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1437e0e-5461-43c3-a7ed-712b006033da-oauth-serving-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.007048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.006889 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbftn\" (UniqueName: \"kubernetes.io/projected/c1437e0e-5461-43c3-a7ed-712b006033da-kube-api-access-rbftn\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.007048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.006899 4227 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1437e0e-5461-43c3-a7ed-712b006033da-console-serving-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.251039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.250968 4227 generic.go:356] "Generic (PLEG): container finished" podID="c1437e0e-5461-43c3-a7ed-712b006033da" containerID="375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e" exitCode=0 Nov 03 20:30:21.251039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.251016 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57df498d94-mcrcw" Nov 03 20:30:21.251229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.251046 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57df498d94-mcrcw" event={"ID":"c1437e0e-5461-43c3-a7ed-712b006033da","Type":"ContainerDied","Data":"375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e"} Nov 03 20:30:21.251229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.251083 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57df498d94-mcrcw" event={"ID":"c1437e0e-5461-43c3-a7ed-712b006033da","Type":"ContainerDied","Data":"6cb58162da746aea527a5f8fe152232b0fe12f95ce6983ba699ed7ca4fcaa487"} Nov 03 20:30:21.251229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.251101 4227 scope.go:122] "RemoveContainer" containerID="375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e" Nov 03 20:30:21.259001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.258982 4227 scope.go:122] "RemoveContainer" containerID="375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e" Nov 03 20:30:21.259260 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:30:21.259241 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e\": container with ID starting with 375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e not found: ID does not exist" containerID="375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e" Nov 03 20:30:21.259317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.259269 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e"} err="failed to get container status \"375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e\": rpc error: code = NotFound desc = could not find container \"375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e\": container with ID starting with 375c7a7dfb757835376a51fc41204e68e743d861359091dfabf40dda967efc1e not found: ID does not exist" Nov 03 20:30:21.403572 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.403549 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57df498d94-mcrcw"] Nov 03 20:30:21.406674 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:21.406656 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57df498d94-mcrcw"] Nov 03 20:30:22.896852 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:30:22.896813 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1437e0e-5461-43c3-a7ed-712b006033da" path="/var/lib/kubelet/pods/c1437e0e-5461-43c3-a7ed-712b006033da/volumes" Nov 03 20:30:26.080577 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:30:26.080540 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:30:56.080241 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:30:56.080205 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:31:11.787747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:11.787652 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:31:11.788253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:11.787999 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="alertmanager" containerID="cri-o://bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7" gracePeriod=120 Nov 03 20:31:11.788253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:11.788065 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-web" containerID="cri-o://8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb" gracePeriod=120 Nov 03 20:31:11.788253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:11.788093 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-metric" containerID="cri-o://c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a" gracePeriod=120 Nov 03 20:31:11.788253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:11.788204 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy" containerID="cri-o://87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049" gracePeriod=120 Nov 03 20:31:11.788253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:11.788209 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="prom-label-proxy" containerID="cri-o://dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734" gracePeriod=120 Nov 03 20:31:11.788487 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:11.788084 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="config-reloader" containerID="cri-o://a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771" gracePeriod=120 Nov 03 20:31:12.423980 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.423946 4227 generic.go:356] "Generic (PLEG): container finished" podID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerID="dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734" exitCode=0 Nov 03 20:31:12.423980 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.423973 4227 generic.go:356] "Generic (PLEG): container finished" podID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerID="c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a" exitCode=0 Nov 03 20:31:12.423980 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.423982 4227 generic.go:356] "Generic (PLEG): container finished" podID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerID="87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049" exitCode=0 Nov 03 20:31:12.423980 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.423988 4227 generic.go:356] "Generic (PLEG): container finished" podID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerID="a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771" exitCode=0 Nov 03 20:31:12.423980 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.423993 4227 generic.go:356] "Generic (PLEG): container finished" podID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerID="bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7" exitCode=0 Nov 03 20:31:12.424375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.424018 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734"} Nov 03 20:31:12.424375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.424095 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a"} Nov 03 20:31:12.424375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.424136 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049"} Nov 03 20:31:12.424375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.424189 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771"} Nov 03 20:31:12.424375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:12.424227 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7"} Nov 03 20:31:13.037051 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.037025 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.184138 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184105 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-main-db\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184196 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184253 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-web\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184282 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-tls-assets\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-tls-assets\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184319 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-metrics-client-ca\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184385 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-kube-api-access-w9jrs\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-kube-api-access-w9jrs\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184422 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-web-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-web-config\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184457 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-main-tls\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184500 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-trusted-ca-bundle\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184513 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-main-db" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:13.184579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184527 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-volume\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-volume\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184871 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184682 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184871 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184719 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-out\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184871 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184753 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-cluster-tls-config\") pod \"c85a35ae-3cc8-47fa-b85a-4d548692c113\" (UID: \"c85a35ae-3cc8-47fa-b85a-4d548692c113\") " Nov 03 20:31:13.184871 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.184816 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-metrics-client-ca" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:13.185060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.185005 4227 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-main-db\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.185060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.185023 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-metrics-client-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.186696 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.186658 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-trusted-ca-bundle" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:13.187105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.187081 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-metric" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:13.187544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.187503 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-tls-assets" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:13.187925 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.187890 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-volume" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:13.188015 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.187915 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-web" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:13.188015 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.187948 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:13.188015 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.187972 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-main-tls" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:13.188652 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.188619 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-kube-api-access-w9jrs" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "kube-api-access-w9jrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:13.188738 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.188672 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-out" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:13.191997 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.191970 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-cluster-tls-config" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:13.197442 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.197420 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-web-config" pod "c85a35ae-3cc8-47fa-b85a-4d548692c113" (UID: "c85a35ae-3cc8-47fa-b85a-4d548692c113"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:13.285349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285318 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jrs\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-kube-api-access-w9jrs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285342 4227 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-web-config\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285353 4227 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-main-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285363 4227 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c85a35ae-3cc8-47fa-b85a-4d548692c113-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285372 4227 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285382 4227 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285390 4227 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c85a35ae-3cc8-47fa-b85a-4d548692c113-config-out\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285398 4227 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-cluster-tls-config\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285407 4227 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285417 4227 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c85a35ae-3cc8-47fa-b85a-4d548692c113-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.285569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.285425 4227 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c85a35ae-3cc8-47fa-b85a-4d548692c113-tls-assets\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:13.431309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.431276 4227 generic.go:356] "Generic (PLEG): container finished" podID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerID="8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb" exitCode=0 Nov 03 20:31:13.431488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.431331 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb"} Nov 03 20:31:13.431488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.431380 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.431488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.431397 4227 scope.go:122] "RemoveContainer" containerID="dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734" Nov 03 20:31:13.431673 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.431387 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c85a35ae-3cc8-47fa-b85a-4d548692c113","Type":"ContainerDied","Data":"2253ec496977edac301b128c5c01dcd50b6ef12b56a62909b8a02bb0e39c5c80"} Nov 03 20:31:13.439581 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.439544 4227 scope.go:122] "RemoveContainer" containerID="c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a" Nov 03 20:31:13.446398 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.446377 4227 scope.go:122] "RemoveContainer" containerID="87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049" Nov 03 20:31:13.452989 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.452970 4227 scope.go:122] "RemoveContainer" containerID="8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb" Nov 03 20:31:13.459234 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.459213 4227 scope.go:122] "RemoveContainer" containerID="a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771" Nov 03 20:31:13.465828 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.465805 4227 scope.go:122] "RemoveContainer" containerID="bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7" Nov 03 20:31:13.472700 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.472670 4227 scope.go:122] "RemoveContainer" containerID="01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5" Nov 03 20:31:13.479104 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.479081 4227 scope.go:122] "RemoveContainer" containerID="dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734" Nov 03 20:31:13.479372 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:13.479354 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734\": container with ID starting with dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734 not found: ID does not exist" containerID="dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734" Nov 03 20:31:13.479432 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.479384 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734"} err="failed to get container status \"dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734\": rpc error: code = NotFound desc = could not find container \"dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734\": container with ID starting with dd959f40f83df105d23c0912f48d2339387677ea9b914c28e079f3ec32d0d734 not found: ID does not exist" Nov 03 20:31:13.479432 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.479401 4227 scope.go:122] "RemoveContainer" containerID="c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a" Nov 03 20:31:13.479641 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:13.479627 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a\": container with ID starting with c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a not found: ID does not exist" containerID="c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a" Nov 03 20:31:13.479682 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.479645 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a"} err="failed to get container status \"c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a\": rpc error: code = NotFound desc = could not find container \"c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a\": container with ID starting with c5c54a8713bee66e2e64729991d2dd14f9a9bd5816ed9907a8183c641c602f9a not found: ID does not exist" Nov 03 20:31:13.479682 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.479657 4227 scope.go:122] "RemoveContainer" containerID="87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049" Nov 03 20:31:13.479859 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:13.479844 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049\": container with ID starting with 87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049 not found: ID does not exist" containerID="87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049" Nov 03 20:31:13.479894 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.479863 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049"} err="failed to get container status \"87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049\": rpc error: code = NotFound desc = could not find container \"87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049\": container with ID starting with 87afc2157ba99aeaf85208ff815de269d706ecd72dd5d8616ce2b0d50fffa049 not found: ID does not exist" Nov 03 20:31:13.479894 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.479876 4227 scope.go:122] "RemoveContainer" containerID="8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb" Nov 03 20:31:13.480104 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:13.480087 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb\": container with ID starting with 8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb not found: ID does not exist" containerID="8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb" Nov 03 20:31:13.480154 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.480107 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb"} err="failed to get container status \"8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb\": rpc error: code = NotFound desc = could not find container \"8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb\": container with ID starting with 8cf4a540d34a74df032436afeff5bfd51b40b4e5353b20feca4d7b62e17b74eb not found: ID does not exist" Nov 03 20:31:13.480154 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.480121 4227 scope.go:122] "RemoveContainer" containerID="a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771" Nov 03 20:31:13.480361 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:13.480345 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771\": container with ID starting with a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771 not found: ID does not exist" containerID="a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771" Nov 03 20:31:13.480412 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.480364 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771"} err="failed to get container status \"a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771\": rpc error: code = NotFound desc = could not find container \"a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771\": container with ID starting with a16a3205a3ff24721136854043fc6141bdd9a700b257e3fae4103418c5df5771 not found: ID does not exist" Nov 03 20:31:13.480412 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.480376 4227 scope.go:122] "RemoveContainer" containerID="bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7" Nov 03 20:31:13.480624 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:13.480594 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7\": container with ID starting with bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7 not found: ID does not exist" containerID="bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7" Nov 03 20:31:13.480624 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.480622 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7"} err="failed to get container status \"bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7\": rpc error: code = NotFound desc = could not find container \"bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7\": container with ID starting with bda9eba5a55ba9e7fb65754e6b0e5fc7d757c2d205cda912d13229a9128f34b7 not found: ID does not exist" Nov 03 20:31:13.480800 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.480638 4227 scope.go:122] "RemoveContainer" containerID="01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5" Nov 03 20:31:13.480911 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:13.480883 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5\": container with ID starting with 01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5 not found: ID does not exist" containerID="01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5" Nov 03 20:31:13.480911 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.480901 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5"} err="failed to get container status \"01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5\": rpc error: code = NotFound desc = could not find container \"01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5\": container with ID starting with 01a9ede4c19ff68d808e574c1f399679f772d5111470e032ed22d175dd4093c5 not found: ID does not exist" Nov 03 20:31:13.587118 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.587089 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:31:13.590698 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.590679 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:31:13.633989 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.633958 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:31:13.634291 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634274 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="config-reloader" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634292 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="config-reloader" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634311 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c1437e0e-5461-43c3-a7ed-712b006033da" containerName="console" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634319 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1437e0e-5461-43c3-a7ed-712b006033da" containerName="console" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634329 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="init-config-reloader" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634335 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="init-config-reloader" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634345 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634352 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634365 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="prom-label-proxy" Nov 03 20:31:13.634380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634372 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="prom-label-proxy" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634393 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-metric" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634401 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-metric" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634413 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="alertmanager" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634421 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="alertmanager" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634432 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-web" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634439 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-web" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634504 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="config-reloader" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634515 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-metric" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634525 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="alertmanager" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634534 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1437e0e-5461-43c3-a7ed-712b006033da" containerName="console" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634543 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy-web" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634551 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="kube-rbac-proxy" Nov 03 20:31:13.634802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.634561 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" containerName="prom-label-proxy" Nov 03 20:31:13.638675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.638655 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.688368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688303 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 20:31:13.688368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688317 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l6mwl\"" Nov 03 20:31:13.688368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688352 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 20:31:13.688676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688395 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 20:31:13.688676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688402 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 20:31:13.688676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688427 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 20:31:13.688676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688532 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 20:31:13.688676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688644 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 20:31:13.688852 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.688659 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 20:31:13.734763 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.734696 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 20:31:13.743063 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.743035 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:31:13.789727 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789684 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-web-config\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.789727 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789727 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/005f73b8-626f-443f-a6c5-033d0d1be875-tls-assets\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.789935 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789748 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/005f73b8-626f-443f-a6c5-033d0d1be875-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.789935 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789877 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.789935 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789907 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790026 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789934 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005f73b8-626f-443f-a6c5-033d0d1be875-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790026 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789977 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/005f73b8-626f-443f-a6c5-033d0d1be875-config-out\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790026 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.789998 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-config-volume\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790026 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.790013 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790138 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.790058 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq52\" (UniqueName: \"kubernetes.io/projected/005f73b8-626f-443f-a6c5-033d0d1be875-kube-api-access-khq52\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790138 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.790115 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.790180 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.790235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.790205 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/005f73b8-626f-443f-a6c5-033d0d1be875-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.890948 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/005f73b8-626f-443f-a6c5-033d0d1be875-config-out\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891009 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-config-volume\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891034 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891062 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq52\" (UniqueName: \"kubernetes.io/projected/005f73b8-626f-443f-a6c5-033d0d1be875-kube-api-access-khq52\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891089 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891250 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891468 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891308 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/005f73b8-626f-443f-a6c5-033d0d1be875-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891468 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891354 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-web-config\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891468 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891380 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/005f73b8-626f-443f-a6c5-033d0d1be875-tls-assets\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891468 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891403 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/005f73b8-626f-443f-a6c5-033d0d1be875-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891473 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891508 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891535 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005f73b8-626f-443f-a6c5-033d0d1be875-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.891740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.891668 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/005f73b8-626f-443f-a6c5-033d0d1be875-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.892704 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.892364 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/005f73b8-626f-443f-a6c5-033d0d1be875-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.892704 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.892518 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005f73b8-626f-443f-a6c5-033d0d1be875-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.893909 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.893880 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/005f73b8-626f-443f-a6c5-033d0d1be875-config-out\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.894262 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.894229 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.894262 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.894248 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-config-volume\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.894612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.894588 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.895141 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.895116 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-web-config\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.895242 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.895148 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/005f73b8-626f-443f-a6c5-033d0d1be875-tls-assets\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.895356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.895329 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.895607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.895585 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.895832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.895815 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/005f73b8-626f-443f-a6c5-033d0d1be875-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.944825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.944790 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq52\" (UniqueName: \"kubernetes.io/projected/005f73b8-626f-443f-a6c5-033d0d1be875-kube-api-access-khq52\") pod \"alertmanager-main-0\" (UID: \"005f73b8-626f-443f-a6c5-033d0d1be875\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:13.958759 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:13.958726 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:31:14.379716 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:14.379633 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:31:14.388509 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:31:14.388481 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005f73b8_626f_443f_a6c5_033d0d1be875.slice/crio-408551414fa6a7c92b03637b415a71c2342aaeb33fb25f05f2622acd10d32328 WatchSource:0}: Error finding container 408551414fa6a7c92b03637b415a71c2342aaeb33fb25f05f2622acd10d32328: Status 404 returned error can't find the container with id 408551414fa6a7c92b03637b415a71c2342aaeb33fb25f05f2622acd10d32328 Nov 03 20:31:14.435626 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:14.435599 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerStarted","Data":"408551414fa6a7c92b03637b415a71c2342aaeb33fb25f05f2622acd10d32328"} Nov 03 20:31:14.897493 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:14.897463 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85a35ae-3cc8-47fa-b85a-4d548692c113" path="/var/lib/kubelet/pods/c85a35ae-3cc8-47fa-b85a-4d548692c113/volumes" Nov 03 20:31:15.441419 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:15.441389 4227 generic.go:356] "Generic (PLEG): container finished" podID="005f73b8-626f-443f-a6c5-033d0d1be875" containerID="59ee85e96829c0776c2a677c86a6bddadc0015444eadbd38f157bfbc1fe73d51" exitCode=0 Nov 03 20:31:15.441811 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:15.441464 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerDied","Data":"59ee85e96829c0776c2a677c86a6bddadc0015444eadbd38f157bfbc1fe73d51"} Nov 03 20:31:16.448629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:16.448597 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerStarted","Data":"fe3506c48948cd0c074966b0c52aaafa724809d7191e815039fec32b934b80a7"} Nov 03 20:31:16.449024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:16.448662 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerStarted","Data":"1d9996ac5c31187069cc524b06cdd33d116aafc6a47521fa3e364118b7894bce"} Nov 03 20:31:16.449024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:16.448688 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerStarted","Data":"ba1bdb1da46242253142ada54dd2fef5494f1d5b09131746b711e007d7fe4c69"} Nov 03 20:31:16.449024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:16.448712 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerStarted","Data":"6cdf2ed99415393b4dcc2b76cce566320af7eaaa054ed5d219f5195ccaa9f4a7"} Nov 03 20:31:16.449024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:16.448740 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerStarted","Data":"6f80bc64c325321c36a2e33bbafe35f4554eeb8ae67aa8794d14909caa7e987e"} Nov 03 20:31:16.449024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:16.448763 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"005f73b8-626f-443f-a6c5-033d0d1be875","Type":"ContainerStarted","Data":"63c34c2a7af22ca081a9655a382991774ecdae830e15fb395a68dffcaa9c64a6"} Nov 03 20:31:16.563088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:16.563026 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.550179696 podStartE2EDuration="3.563013872s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.834176ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:16.562677474 +0000 UTC m=+143.799655348" watchObservedRunningTime="2026-11-03 20:31:16.563013872 +0000 UTC m=+143.799991745" Nov 03 20:31:26.079603 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:26.079566 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:31:33.941863 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:33.941827 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f44845446-btth6"] Nov 03 20:31:56.080177 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:56.080131 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:31:58.960927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:58.960874 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f44845446-btth6" podUID="c08ef4c9-2973-4c41-be8d-96a8b6ec423d" containerName="console" containerID="cri-o://110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642" gracePeriod=15 Nov 03 20:31:59.199334 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.199311 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:31:59.310211 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310126 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-tmp\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310211 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310173 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-config\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-config\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310211 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310194 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-serving-cert\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310383 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-oauth-config\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310422 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-oauth-serving-cert\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310462 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-trusted-ca-bundle\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310689 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310444 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-tmp" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:59.310689 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310521 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-service-ca\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310689 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310563 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-kube-api-access-9fgk7\" (UniqueName: \"kubernetes.io/projected/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-kube-api-access-9fgk7\") pod \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\" (UID: \"c08ef4c9-2973-4c41-be8d-96a8b6ec423d\") " Nov 03 20:31:59.310689 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310591 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-config" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:59.310890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310780 4227 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-tmp\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.310890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.310820 4227 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-config\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.311470 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.311406 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-trusted-ca-bundle" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:59.312348 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.312315 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-oauth-serving-cert" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:59.312523 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.312473 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-service-ca" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:59.312806 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.312782 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-serving-cert" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:59.312891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.312859 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-oauth-config" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:59.313985 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.313963 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-kube-api-access-9fgk7" pod "c08ef4c9-2973-4c41-be8d-96a8b6ec423d" (UID: "c08ef4c9-2973-4c41-be8d-96a8b6ec423d"). InnerVolumeSpecName "kube-api-access-9fgk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:59.411330 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.411306 4227 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-oauth-config\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.411330 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.411326 4227 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-oauth-serving-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.411466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.411336 4227 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-trusted-ca-bundle\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.411466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.411348 4227 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.411466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.411356 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fgk7\" (UniqueName: \"kubernetes.io/projected/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-kube-api-access-9fgk7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.411466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.411365 4227 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c08ef4c9-2973-4c41-be8d-96a8b6ec423d-console-serving-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:59.594231 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.594171 4227 generic.go:356] "Generic (PLEG): container finished" podID="c08ef4c9-2973-4c41-be8d-96a8b6ec423d" containerID="110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642" exitCode=0 Nov 03 20:31:59.594231 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.594216 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44845446-btth6" Nov 03 20:31:59.594354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.594251 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44845446-btth6" event={"ID":"c08ef4c9-2973-4c41-be8d-96a8b6ec423d","Type":"ContainerDied","Data":"110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642"} Nov 03 20:31:59.594354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.594294 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44845446-btth6" event={"ID":"c08ef4c9-2973-4c41-be8d-96a8b6ec423d","Type":"ContainerDied","Data":"0bd2a0e85b494ac6302a648cccfee97916c3f8470cf008568caa79a59bc51bbe"} Nov 03 20:31:59.594354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.594315 4227 scope.go:122] "RemoveContainer" containerID="110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642" Nov 03 20:31:59.603686 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.603666 4227 scope.go:122] "RemoveContainer" containerID="110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642" Nov 03 20:31:59.604004 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:31:59.603983 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642\": container with ID starting with 110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642 not found: ID does not exist" containerID="110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642" Nov 03 20:31:59.604088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.604015 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642"} err="failed to get container status \"110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642\": rpc error: code = NotFound desc = could not find container \"110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642\": container with ID starting with 110e4ccd7ed01731b8ce548c903d973cb13c2ec07c91301396adbf7a63c58642 not found: ID does not exist" Nov 03 20:31:59.746726 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.746699 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f44845446-btth6"] Nov 03 20:31:59.792982 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:31:59.792957 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f44845446-btth6"] Nov 03 20:32:00.902213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:32:00.902173 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08ef4c9-2973-4c41-be8d-96a8b6ec423d" path="/var/lib/kubelet/pods/c08ef4c9-2973-4c41-be8d-96a8b6ec423d/volumes" Nov 03 20:32:26.079884 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:32:26.079845 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:32:56.080059 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:32:56.080022 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:33:26.080487 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:33:26.080446 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:33:52.908403 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:33:52.908373 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:33:52.908789 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:33:52.908505 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:33:56.079793 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:33:56.079756 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:34:26.081391 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:34:26.081355 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:34:52.608553 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.608518 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl"] Nov 03 20:34:52.608930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.608785 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c08ef4c9-2973-4c41-be8d-96a8b6ec423d" containerName="console" Nov 03 20:34:52.608930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.608794 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08ef4c9-2973-4c41-be8d-96a8b6ec423d" containerName="console" Nov 03 20:34:52.608930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.608849 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08ef4c9-2973-4c41-be8d-96a8b6ec423d" containerName="console" Nov 03 20:34:52.610963 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.610946 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.670330 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.670293 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:34:52.670508 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.670383 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:34:52.671934 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.671905 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl"] Nov 03 20:34:52.706232 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.706205 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-chl6n\"" Nov 03 20:34:52.714596 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.714573 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.714706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.714616 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqjcr\" (UniqueName: \"kubernetes.io/projected/f4b7fec9-84a9-47a0-913b-45192de19e7e-kube-api-access-xqjcr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.714706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.714652 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.815450 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.815408 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.815635 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.815455 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqjcr\" (UniqueName: \"kubernetes.io/projected/f4b7fec9-84a9-47a0-913b-45192de19e7e-kube-api-access-xqjcr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.815635 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.815486 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.815805 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.815774 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.815882 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.815834 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:52.830911 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.830889 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h"] Nov 03 20:34:52.833257 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.833241 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.886630 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.886556 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:34:52.889489 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.889458 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:34:52.889583 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.889529 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:34:52.890298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.890280 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:34:52.890298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.890282 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:34:52.890298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.890296 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:34:52.890451 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.890321 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4mvmz\"" Nov 03 20:34:52.899365 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.899342 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h"] Nov 03 20:34:52.915553 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.915528 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:34:52.916860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.916834 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.916967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.916879 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.917008 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.916979 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.917052 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.917012 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.917052 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.917037 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f593646-8eec-4531-9c13-8e71722d5a9b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.917122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.917058 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.917122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.917077 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7v6f\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-kube-api-access-s7v6f\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:52.966828 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.966801 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:34:52.972199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:52.972153 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqjcr\" (UniqueName: \"kubernetes.io/projected/f4b7fec9-84a9-47a0-913b-45192de19e7e-kube-api-access-xqjcr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:53.018089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018061 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.018089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018092 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f593646-8eec-4531-9c13-8e71722d5a9b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.018282 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018117 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.018282 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018140 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7v6f\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-kube-api-access-s7v6f\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.018282 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018242 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.018282 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018272 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.018478 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018309 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.018899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.018873 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.020577 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.020551 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.020577 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.020563 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f593646-8eec-4531-9c13-8e71722d5a9b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.020702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.020628 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.020738 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.020708 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.072699 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.072665 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.072877 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.072854 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7v6f\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-kube-api-access-s7v6f\") pod \"istiod-openshift-gateway-7876bfcb74-b5q5h\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.143177 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.143087 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:53.272364 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.272321 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-chl6n\"" Nov 03 20:34:53.274366 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.274348 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:34:53.560326 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.560287 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h"] Nov 03 20:34:53.568624 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:34:53.568596 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f593646_8eec_4531_9c13_8e71722d5a9b.slice/crio-7ea6cd4c9e19a603a0f529f84fe3197a16ac53600d9c1827057471d4e1bd688b WatchSource:0}: Error finding container 7ea6cd4c9e19a603a0f529f84fe3197a16ac53600d9c1827057471d4e1bd688b: Status 404 returned error can't find the container with id 7ea6cd4c9e19a603a0f529f84fe3197a16ac53600d9c1827057471d4e1bd688b Nov 03 20:34:53.570510 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.570493 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:34:53.673611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:53.673576 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl"] Nov 03 20:34:53.681112 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:34:53.681085 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b7fec9_84a9_47a0_913b_45192de19e7e.slice/crio-f19479203365e6616501306d8c3100fe77a1b89172e0f6d695875134ba751bc5 WatchSource:0}: Error finding container f19479203365e6616501306d8c3100fe77a1b89172e0f6d695875134ba751bc5: Status 404 returned error can't find the container with id f19479203365e6616501306d8c3100fe77a1b89172e0f6d695875134ba751bc5 Nov 03 20:34:54.141372 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:54.141330 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" event={"ID":"f4b7fec9-84a9-47a0-913b-45192de19e7e","Type":"ContainerStarted","Data":"f19479203365e6616501306d8c3100fe77a1b89172e0f6d695875134ba751bc5"} Nov 03 20:34:54.142404 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:54.142380 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" event={"ID":"6f593646-8eec-4531-9c13-8e71722d5a9b","Type":"ContainerStarted","Data":"7ea6cd4c9e19a603a0f529f84fe3197a16ac53600d9c1827057471d4e1bd688b"} Nov 03 20:34:56.083084 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:34:56.083029 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:34:58.219592 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:58.219561 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:34:58.219988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:58.219622 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:34:59.165625 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:59.165586 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" event={"ID":"6f593646-8eec-4531-9c13-8e71722d5a9b","Type":"ContainerStarted","Data":"8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161"} Nov 03 20:34:59.166006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:59.165979 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:34:59.167870 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:59.167839 4227 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-b5q5h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:34:59.167986 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:59.167897 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" podUID="6f593646-8eec-4531-9c13-8e71722d5a9b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:34:59.358903 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:34:59.358851 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" podStartSLOduration=2.710741333 podStartE2EDuration="7.358835798s" totalImagesPullingTime="4.648094465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:59.321138919 +0000 UTC m=+366.558116792" watchObservedRunningTime="2026-11-03 20:34:59.358835798 +0000 UTC m=+366.595813672" Nov 03 20:35:00.169843 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:00.169811 4227 generic.go:356] "Generic (PLEG): container finished" podID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerID="fd9fb567935a95757d1ca516b3b8f14b6d07a762ae6e6fda7adb4f92a383a876" exitCode=0 Nov 03 20:35:00.169989 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:00.169902 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" event={"ID":"f4b7fec9-84a9-47a0-913b-45192de19e7e","Type":"ContainerDied","Data":"fd9fb567935a95757d1ca516b3b8f14b6d07a762ae6e6fda7adb4f92a383a876"} Nov 03 20:35:00.170715 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:00.170689 4227 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-b5q5h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:35:00.170828 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:00.170770 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" podUID="6f593646-8eec-4531-9c13-8e71722d5a9b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:02.178986 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:02.178947 4227 generic.go:356] "Generic (PLEG): container finished" podID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerID="7f9f5d74b575e185b2178f11ab3f44ce4764f3bf9e213bf55220d1abd9729e18" exitCode=0 Nov 03 20:35:02.179448 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:02.179031 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" event={"ID":"f4b7fec9-84a9-47a0-913b-45192de19e7e","Type":"ContainerDied","Data":"7f9f5d74b575e185b2178f11ab3f44ce4764f3bf9e213bf55220d1abd9729e18"} Nov 03 20:35:03.170772 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:03.170743 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:35:09.207747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:09.207711 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" event={"ID":"f4b7fec9-84a9-47a0-913b-45192de19e7e","Type":"ContainerStarted","Data":"542f232306150c84c7f2ebfdd818ffb345979878d04006d050f9be3c4cb333d2"} Nov 03 20:35:09.316319 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:09.316263 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" podStartSLOduration=3.366702619 podStartE2EDuration="17.316247095s" totalImagesPullingTime="13.949544476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:09.315469523 +0000 UTC m=+376.552447398" watchObservedRunningTime="2026-11-03 20:35:09.316247095 +0000 UTC m=+376.553224969" Nov 03 20:35:10.213118 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:10.213084 4227 generic.go:356] "Generic (PLEG): container finished" podID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerID="542f232306150c84c7f2ebfdd818ffb345979878d04006d050f9be3c4cb333d2" exitCode=0 Nov 03 20:35:10.213507 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:10.213143 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" event={"ID":"f4b7fec9-84a9-47a0-913b-45192de19e7e","Type":"ContainerDied","Data":"542f232306150c84c7f2ebfdd818ffb345979878d04006d050f9be3c4cb333d2"} Nov 03 20:35:11.339788 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.339763 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:35:11.357541 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.357517 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-util\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-util\") pod \"f4b7fec9-84a9-47a0-913b-45192de19e7e\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " Nov 03 20:35:11.357610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.357596 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b7fec9-84a9-47a0-913b-45192de19e7e-kube-api-access-xqjcr\" (UniqueName: \"kubernetes.io/projected/f4b7fec9-84a9-47a0-913b-45192de19e7e-kube-api-access-xqjcr\") pod \"f4b7fec9-84a9-47a0-913b-45192de19e7e\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " Nov 03 20:35:11.357649 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.357622 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-bundle\") pod \"f4b7fec9-84a9-47a0-913b-45192de19e7e\" (UID: \"f4b7fec9-84a9-47a0-913b-45192de19e7e\") " Nov 03 20:35:11.358115 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.358097 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-bundle" pod "f4b7fec9-84a9-47a0-913b-45192de19e7e" (UID: "f4b7fec9-84a9-47a0-913b-45192de19e7e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:11.359611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.359589 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b7fec9-84a9-47a0-913b-45192de19e7e-kube-api-access-xqjcr" pod "f4b7fec9-84a9-47a0-913b-45192de19e7e" (UID: "f4b7fec9-84a9-47a0-913b-45192de19e7e"). InnerVolumeSpecName "kube-api-access-xqjcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:11.363131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.363107 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-util" pod "f4b7fec9-84a9-47a0-913b-45192de19e7e" (UID: "f4b7fec9-84a9-47a0-913b-45192de19e7e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:11.458480 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.458449 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqjcr\" (UniqueName: \"kubernetes.io/projected/f4b7fec9-84a9-47a0-913b-45192de19e7e-kube-api-access-xqjcr\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.458480 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.458473 4227 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-bundle\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.458480 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:11.458483 4227 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b7fec9-84a9-47a0-913b-45192de19e7e-util\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:12.220988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:12.220954 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" event={"ID":"f4b7fec9-84a9-47a0-913b-45192de19e7e","Type":"ContainerDied","Data":"f19479203365e6616501306d8c3100fe77a1b89172e0f6d695875134ba751bc5"} Nov 03 20:35:12.220988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:12.220992 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712sctl" Nov 03 20:35:12.221211 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:12.220998 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19479203365e6616501306d8c3100fe77a1b89172e0f6d695875134ba751bc5" Nov 03 20:35:26.079294 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:35:26.079255 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:35:36.304471 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304395 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl"] Nov 03 20:35:36.304927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304660 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerName="pull" Nov 03 20:35:36.304927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304670 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerName="pull" Nov 03 20:35:36.304927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304689 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerName="extract" Nov 03 20:35:36.304927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304695 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerName="extract" Nov 03 20:35:36.304927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304706 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerName="util" Nov 03 20:35:36.304927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304711 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerName="util" Nov 03 20:35:36.304927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.304766 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b7fec9-84a9-47a0-913b-45192de19e7e" containerName="extract" Nov 03 20:35:36.307020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.307000 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.358388 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.358364 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:35:36.358919 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.358903 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:35:36.363919 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.363902 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:35:36.404464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.404447 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:35:36.404905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.404869 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-s65qj\"" Nov 03 20:35:36.404905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.404873 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:35:36.427524 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.427499 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl"] Nov 03 20:35:36.431429 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.431332 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.431429 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.431386 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.431564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.431478 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5c53545-4051-4bd4-82ec-418196444507-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.431564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.431525 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.431564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.431557 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h645\" (UniqueName: \"kubernetes.io/projected/d5c53545-4051-4bd4-82ec-418196444507-kube-api-access-8h645\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.431747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.431600 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c53545-4051-4bd4-82ec-418196444507-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.531895 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.531868 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.532029 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.531902 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5c53545-4051-4bd4-82ec-418196444507-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.532029 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.531923 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.532121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.532032 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h645\" (UniqueName: \"kubernetes.io/projected/d5c53545-4051-4bd4-82ec-418196444507-kube-api-access-8h645\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.532121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.532086 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c53545-4051-4bd4-82ec-418196444507-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.532249 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.532171 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.532302 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.532277 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5c53545-4051-4bd4-82ec-418196444507-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.534355 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.534329 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c53545-4051-4bd4-82ec-418196444507-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.534521 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.534500 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.534562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.534504 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.534562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.534507 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5c53545-4051-4bd4-82ec-418196444507-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.586683 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.586630 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h645\" (UniqueName: \"kubernetes.io/projected/d5c53545-4051-4bd4-82ec-418196444507-kube-api-access-8h645\") pod \"opendatahub-operator-controller-manager-768646756f-x4xvl\" (UID: \"d5c53545-4051-4bd4-82ec-418196444507\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:36.619262 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:36.619240 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:37.105058 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:37.105028 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl"] Nov 03 20:35:37.112405 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:35:37.112378 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c53545_4051_4bd4_82ec_418196444507.slice/crio-20e0ae9c294a7a0402ddad4f22dddf129daac098efb7e155526d04b547e30ee7 WatchSource:0}: Error finding container 20e0ae9c294a7a0402ddad4f22dddf129daac098efb7e155526d04b547e30ee7: Status 404 returned error can't find the container with id 20e0ae9c294a7a0402ddad4f22dddf129daac098efb7e155526d04b547e30ee7 Nov 03 20:35:37.304182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:37.304140 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" event={"ID":"d5c53545-4051-4bd4-82ec-418196444507","Type":"ContainerStarted","Data":"20e0ae9c294a7a0402ddad4f22dddf129daac098efb7e155526d04b547e30ee7"} Nov 03 20:35:41.319493 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:41.319413 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" event={"ID":"d5c53545-4051-4bd4-82ec-418196444507","Type":"ContainerStarted","Data":"981d76aab5c2ce24fa237e28bdc63798c1888d28a7d1402c4cab990e83755aee"} Nov 03 20:35:42.324858 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:42.324818 4227 generic.go:356] "Generic (PLEG): container finished" podID="d5c53545-4051-4bd4-82ec-418196444507" containerID="981d76aab5c2ce24fa237e28bdc63798c1888d28a7d1402c4cab990e83755aee" exitCode=0 Nov 03 20:35:42.325317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:42.324871 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" event={"ID":"d5c53545-4051-4bd4-82ec-418196444507","Type":"ContainerDied","Data":"981d76aab5c2ce24fa237e28bdc63798c1888d28a7d1402c4cab990e83755aee"} Nov 03 20:35:44.332766 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:44.332734 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" event={"ID":"d5c53545-4051-4bd4-82ec-418196444507","Type":"ContainerStarted","Data":"8597282a4088fb7f250d66715cad2a65f9a7999f6adf65d2684d33b4d49adcc6"} Nov 03 20:35:44.333247 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:44.332825 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:44.636394 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:44.636285 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" podStartSLOduration=3.376863367 podStartE2EDuration="8.636270559s" totalImagesPullingTime="4.841703833s" totalInitContainerRuntime="417.703359ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:44.588715348 +0000 UTC m=+411.825693221" watchObservedRunningTime="2026-11-03 20:35:44.636270559 +0000 UTC m=+411.873248433" Nov 03 20:35:55.339110 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:35:55.339075 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-x4xvl" Nov 03 20:35:56.079436 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:35:56.079400 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:36:16.178717 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.178687 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg"] Nov 03 20:36:16.185315 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.185294 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.235938 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.235906 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg"] Nov 03 20:36:16.281074 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.281043 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:36:16.281229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.281087 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:36:16.281229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.281138 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ccl98\"" Nov 03 20:36:16.327722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.327692 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-tmp\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.327867 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.327803 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxvh\" (UniqueName: \"kubernetes.io/projected/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-kube-api-access-7lxvh\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.327867 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.327846 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-tls-certs\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.429243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.429173 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxvh\" (UniqueName: \"kubernetes.io/projected/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-kube-api-access-7lxvh\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.429243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.429225 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-tls-certs\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.429411 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.429290 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-tmp\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.431715 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.431686 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-tls-certs\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.434650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.434625 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-tmp\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.484388 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.484366 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxvh\" (UniqueName: \"kubernetes.io/projected/dba92d2a-f1bf-4af8-ad77-4918b95e6ad6-kube-api-access-7lxvh\") pod \"kube-auth-proxy-bb87d7467-f6vbg\" (UID: \"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6\") " pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.495979 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.495958 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" Nov 03 20:36:16.872271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:16.872126 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg"] Nov 03 20:36:16.879751 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:36:16.879701 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba92d2a_f1bf_4af8_ad77_4918b95e6ad6.slice/crio-d579ca922366ab2ed3e974dbaa947fb30e44ded0c70d4e69c5bca68a7c695ad3 WatchSource:0}: Error finding container d579ca922366ab2ed3e974dbaa947fb30e44ded0c70d4e69c5bca68a7c695ad3: Status 404 returned error can't find the container with id d579ca922366ab2ed3e974dbaa947fb30e44ded0c70d4e69c5bca68a7c695ad3 Nov 03 20:36:17.447404 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.447371 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" event={"ID":"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6","Type":"ContainerStarted","Data":"d579ca922366ab2ed3e974dbaa947fb30e44ded0c70d4e69c5bca68a7c695ad3"} Nov 03 20:36:17.629101 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.629069 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls"] Nov 03 20:36:17.632476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.632461 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:17.646378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.646355 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls"] Nov 03 20:36:17.691620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.691594 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:36:17.691815 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.691798 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jjnk7\"" Nov 03 20:36:17.691870 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.691815 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:36:17.739183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.739124 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7g8ls\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:17.739183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.739173 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqls\" (UniqueName: \"kubernetes.io/projected/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-kube-api-access-4sqls\") pod \"kuadrant-operator-controller-manager-678d87766f-7g8ls\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:17.840414 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.840388 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7g8ls\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:17.840514 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.840419 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqls\" (UniqueName: \"kubernetes.io/projected/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-kube-api-access-4sqls\") pod \"kuadrant-operator-controller-manager-678d87766f-7g8ls\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:17.840759 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.840741 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7g8ls\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:17.897276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.897253 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqls\" (UniqueName: \"kubernetes.io/projected/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-kube-api-access-4sqls\") pod \"kuadrant-operator-controller-manager-678d87766f-7g8ls\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:17.944621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:17.944599 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:18.415545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:18.415513 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls"] Nov 03 20:36:18.423982 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:36:18.423945 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88e2c41_c43e_4d07_bcee_ca2a13039ab7.slice/crio-29fec6e4a04fd2d7944a08e13a6e517c136f76113fe31329d6bc4d6be3d02651 WatchSource:0}: Error finding container 29fec6e4a04fd2d7944a08e13a6e517c136f76113fe31329d6bc4d6be3d02651: Status 404 returned error can't find the container with id 29fec6e4a04fd2d7944a08e13a6e517c136f76113fe31329d6bc4d6be3d02651 Nov 03 20:36:18.452996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:18.452969 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" event={"ID":"c88e2c41-c43e-4d07-bcee-ca2a13039ab7","Type":"ContainerStarted","Data":"29fec6e4a04fd2d7944a08e13a6e517c136f76113fe31329d6bc4d6be3d02651"} Nov 03 20:36:22.471646 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:22.471606 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" event={"ID":"dba92d2a-f1bf-4af8-ad77-4918b95e6ad6","Type":"ContainerStarted","Data":"6c7cd277ace1f3f1dc0b762f2c1cd3676dcdd96db04afcb87c156d4cbd02c410"} Nov 03 20:36:22.580567 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:22.580518 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-bb87d7467-f6vbg" podStartSLOduration=1.641205588 podStartE2EDuration="6.580504654s" totalImagesPullingTime="4.939299066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:22.579503224 +0000 UTC m=+449.816481100" watchObservedRunningTime="2026-11-03 20:36:22.580504654 +0000 UTC m=+449.817482522" Nov 03 20:36:25.485491 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:25.485450 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" event={"ID":"c88e2c41-c43e-4d07-bcee-ca2a13039ab7","Type":"ContainerStarted","Data":"34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1"} Nov 03 20:36:25.485862 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:25.485559 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:25.686632 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:25.686585 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" podStartSLOduration=1.84079651 podStartE2EDuration="8.68657278s" totalImagesPullingTime="6.84577627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:25.639652112 +0000 UTC m=+452.876629986" watchObservedRunningTime="2026-11-03 20:36:25.68657278 +0000 UTC m=+452.923550649" Nov 03 20:36:26.081474 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:36:26.081391 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:36:36.491673 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:36:36.491646 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 20:36:56.079245 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:36:56.079209 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:37:26.079821 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:37:26.079781 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:37:26.222458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.222426 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn"] Nov 03 20:37:26.224775 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.224760 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.338146 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.338076 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn"] Nov 03 20:37:26.344822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.344791 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfwc\" (UniqueName: \"kubernetes.io/projected/32c44223-9bfe-4f28-b55e-290e1f9f7baf-kube-api-access-dcfwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.344949 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.344831 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.344949 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.344861 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.345029 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.344982 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.345064 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.345029 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.345103 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.345094 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c44223-9bfe-4f28-b55e-290e1f9f7baf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.345140 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.345116 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.446336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.446312 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c44223-9bfe-4f28-b55e-290e1f9f7baf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.446451 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.446347 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.446451 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.446380 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfwc\" (UniqueName: \"kubernetes.io/projected/32c44223-9bfe-4f28-b55e-290e1f9f7baf-kube-api-access-dcfwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.446451 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.446412 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.446451 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.446442 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.446653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.446468 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.446653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.446494 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.447229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.447195 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.448713 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.448684 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.448713 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.448706 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c44223-9bfe-4f28-b55e-290e1f9f7baf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.448855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.448691 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.448855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.448762 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32c44223-9bfe-4f28-b55e-290e1f9f7baf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.499395 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.499373 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfwc\" (UniqueName: \"kubernetes.io/projected/32c44223-9bfe-4f28-b55e-290e1f9f7baf-kube-api-access-dcfwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.499559 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.499543 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c44223-9bfe-4f28-b55e-290e1f9f7baf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twmjn\" (UID: \"32c44223-9bfe-4f28-b55e-290e1f9f7baf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.535466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.535443 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:26.952386 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.952352 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn"] Nov 03 20:37:26.959459 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:37:26.959428 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c44223_9bfe_4f28_b55e_290e1f9f7baf.slice/crio-7f43c333ba8ac2afce1c5856575261e9ecf9ef6f1ca8578e4402f19dbf1c42b6 WatchSource:0}: Error finding container 7f43c333ba8ac2afce1c5856575261e9ecf9ef6f1ca8578e4402f19dbf1c42b6: Status 404 returned error can't find the container with id 7f43c333ba8ac2afce1c5856575261e9ecf9ef6f1ca8578e4402f19dbf1c42b6 Nov 03 20:37:26.961360 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.961330 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:26.961452 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:26.961400 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:27.719906 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:27.719868 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" event={"ID":"32c44223-9bfe-4f28-b55e-290e1f9f7baf","Type":"ContainerStarted","Data":"6d9a9bcd6a6ace7c55911962c5aaeacfb1eecad098de52279504b0b03ad5a4d5"} Nov 03 20:37:27.720283 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:27.719915 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" event={"ID":"32c44223-9bfe-4f28-b55e-290e1f9f7baf","Type":"ContainerStarted","Data":"7f43c333ba8ac2afce1c5856575261e9ecf9ef6f1ca8578e4402f19dbf1c42b6"} Nov 03 20:37:27.720283 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:27.720008 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:28.725915 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:28.725889 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" Nov 03 20:37:28.928677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:28.928619 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twmjn" podStartSLOduration=2.9286026720000002 podStartE2EDuration="2.928602672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:27.92340717 +0000 UTC m=+515.160385042" watchObservedRunningTime="2026-11-03 20:37:28.928602672 +0000 UTC m=+516.165580549" Nov 03 20:37:28.930492 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:28.930459 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h"] Nov 03 20:37:28.930647 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:28.930625 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" podUID="6f593646-8eec-4531-9c13-8e71722d5a9b" containerName="discovery" containerID="cri-o://8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161" gracePeriod=30 Nov 03 20:37:29.162803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.162780 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:37:29.269336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269253 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-dns-cert\") pod \"6f593646-8eec-4531-9c13-8e71722d5a9b\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " Nov 03 20:37:29.269336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269300 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f593646-8eec-4531-9c13-8e71722d5a9b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f593646-8eec-4531-9c13-8e71722d5a9b-local-certs\") pod \"6f593646-8eec-4531-9c13-8e71722d5a9b\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " Nov 03 20:37:29.269336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269324 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-cacerts\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-cacerts\") pod \"6f593646-8eec-4531-9c13-8e71722d5a9b\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " Nov 03 20:37:29.269627 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269363 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-kube-api-access-s7v6f\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-kube-api-access-s7v6f\") pod \"6f593646-8eec-4531-9c13-8e71722d5a9b\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " Nov 03 20:37:29.269627 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269384 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-token\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-token\") pod \"6f593646-8eec-4531-9c13-8e71722d5a9b\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " Nov 03 20:37:29.269627 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269466 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-ca-configmap\") pod \"6f593646-8eec-4531-9c13-8e71722d5a9b\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " Nov 03 20:37:29.269627 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269494 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-kubeconfig\") pod \"6f593646-8eec-4531-9c13-8e71722d5a9b\" (UID: \"6f593646-8eec-4531-9c13-8e71722d5a9b\") " Nov 03 20:37:29.269890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.269861 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-ca-configmap" pod "6f593646-8eec-4531-9c13-8e71722d5a9b" (UID: "6f593646-8eec-4531-9c13-8e71722d5a9b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:29.272686 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.272645 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-kubeconfig" pod "6f593646-8eec-4531-9c13-8e71722d5a9b" (UID: "6f593646-8eec-4531-9c13-8e71722d5a9b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:29.272686 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.272657 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f593646-8eec-4531-9c13-8e71722d5a9b-local-certs" pod "6f593646-8eec-4531-9c13-8e71722d5a9b" (UID: "6f593646-8eec-4531-9c13-8e71722d5a9b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:29.272686 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.272676 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-dns-cert" pod "6f593646-8eec-4531-9c13-8e71722d5a9b" (UID: "6f593646-8eec-4531-9c13-8e71722d5a9b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:29.272884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.272656 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-kube-api-access-s7v6f" pod "6f593646-8eec-4531-9c13-8e71722d5a9b" (UID: "6f593646-8eec-4531-9c13-8e71722d5a9b"). InnerVolumeSpecName "kube-api-access-s7v6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:29.272884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.272715 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-token" pod "6f593646-8eec-4531-9c13-8e71722d5a9b" (UID: "6f593646-8eec-4531-9c13-8e71722d5a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:29.272884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.272744 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-cacerts" pod "6f593646-8eec-4531-9c13-8e71722d5a9b" (UID: "6f593646-8eec-4531-9c13-8e71722d5a9b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:29.370360 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.370332 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7v6f\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-kube-api-access-s7v6f\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:29.370474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.370361 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:29.370474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.370376 4227 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-ca-configmap\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:29.370474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.370388 4227 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-kubeconfig\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:29.370474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.370400 4227 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-istio-csr-dns-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:29.370474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.370412 4227 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f593646-8eec-4531-9c13-8e71722d5a9b-local-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:29.370474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.370424 4227 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f593646-8eec-4531-9c13-8e71722d5a9b-cacerts\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:29.729668 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.729634 4227 generic.go:356] "Generic (PLEG): container finished" podID="6f593646-8eec-4531-9c13-8e71722d5a9b" containerID="8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161" exitCode=0 Nov 03 20:37:29.730143 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.729691 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" Nov 03 20:37:29.730143 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.729721 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" event={"ID":"6f593646-8eec-4531-9c13-8e71722d5a9b","Type":"ContainerDied","Data":"8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161"} Nov 03 20:37:29.730143 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.729774 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h" event={"ID":"6f593646-8eec-4531-9c13-8e71722d5a9b","Type":"ContainerDied","Data":"7ea6cd4c9e19a603a0f529f84fe3197a16ac53600d9c1827057471d4e1bd688b"} Nov 03 20:37:29.730143 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.729794 4227 scope.go:122] "RemoveContainer" containerID="8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161" Nov 03 20:37:29.737967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.737946 4227 scope.go:122] "RemoveContainer" containerID="8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161" Nov 03 20:37:29.738245 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:37:29.738228 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161\": container with ID starting with 8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161 not found: ID does not exist" containerID="8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161" Nov 03 20:37:29.738309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.738255 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161"} err="failed to get container status \"8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161\": rpc error: code = NotFound desc = could not find container \"8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161\": container with ID starting with 8efb8cdb3cae16b79fa9eb557314be44edf029a254bf5be23d7758e57da06161 not found: ID does not exist" Nov 03 20:37:29.904978 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.904952 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h"] Nov 03 20:37:29.951520 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:29.951496 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5q5h"] Nov 03 20:37:30.897674 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:30.897643 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f593646-8eec-4531-9c13-8e71722d5a9b" path="/var/lib/kubelet/pods/6f593646-8eec-4531-9c13-8e71722d5a9b/volumes" Nov 03 20:37:32.037640 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.037592 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2"] Nov 03 20:37:32.038072 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.037898 4227 cpu_manager.go:414] "containerMap: removing container" podUID="6f593646-8eec-4531-9c13-8e71722d5a9b" containerName="discovery" Nov 03 20:37:32.038072 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.037909 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f593646-8eec-4531-9c13-8e71722d5a9b" containerName="discovery" Nov 03 20:37:32.038072 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.037974 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f593646-8eec-4531-9c13-8e71722d5a9b" containerName="discovery" Nov 03 20:37:32.041501 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.041480 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.094291 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.094264 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2"] Nov 03 20:37:32.107052 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.107029 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:37:32.139274 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.139256 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-z7sjf\"" Nov 03 20:37:32.139381 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.139363 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:37:32.139428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.139363 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 20:37:32.144132 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.144113 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xjx5n"] Nov 03 20:37:32.146593 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.146579 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.191404 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.191385 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fqwn\" (UniqueName: \"kubernetes.io/projected/09750f2d-31c5-40e7-bd26-deb825796324-kube-api-access-2fqwn\") pod \"llmisvc-controller-manager-869d5dcc5d-4hvg2\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.191487 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.191432 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09750f2d-31c5-40e7-bd26-deb825796324-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4hvg2\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.195403 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.195387 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rb65f\"" Nov 03 20:37:32.195474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.195460 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 20:37:32.200657 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.200637 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xjx5n"] Nov 03 20:37:32.292470 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.292412 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fqwn\" (UniqueName: \"kubernetes.io/projected/09750f2d-31c5-40e7-bd26-deb825796324-kube-api-access-2fqwn\") pod \"llmisvc-controller-manager-869d5dcc5d-4hvg2\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.292470 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.292445 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v92f\" (UniqueName: \"kubernetes.io/projected/41e62745-afd6-45e5-b3fe-4b9c82669d5a-kube-api-access-4v92f\") pod \"model-serving-api-75bd89fdfc-xjx5n\" (UID: \"41e62745-afd6-45e5-b3fe-4b9c82669d5a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.292602 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.292488 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e62745-afd6-45e5-b3fe-4b9c82669d5a-tls-certs\") pod \"model-serving-api-75bd89fdfc-xjx5n\" (UID: \"41e62745-afd6-45e5-b3fe-4b9c82669d5a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.292602 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.292509 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09750f2d-31c5-40e7-bd26-deb825796324-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4hvg2\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.294770 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.294747 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09750f2d-31c5-40e7-bd26-deb825796324-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4hvg2\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.345201 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.345171 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fqwn\" (UniqueName: \"kubernetes.io/projected/09750f2d-31c5-40e7-bd26-deb825796324-kube-api-access-2fqwn\") pod \"llmisvc-controller-manager-869d5dcc5d-4hvg2\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.352141 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.352124 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:32.393638 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.393607 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v92f\" (UniqueName: \"kubernetes.io/projected/41e62745-afd6-45e5-b3fe-4b9c82669d5a-kube-api-access-4v92f\") pod \"model-serving-api-75bd89fdfc-xjx5n\" (UID: \"41e62745-afd6-45e5-b3fe-4b9c82669d5a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.393771 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.393675 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e62745-afd6-45e5-b3fe-4b9c82669d5a-tls-certs\") pod \"model-serving-api-75bd89fdfc-xjx5n\" (UID: \"41e62745-afd6-45e5-b3fe-4b9c82669d5a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.399115 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.398726 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e62745-afd6-45e5-b3fe-4b9c82669d5a-tls-certs\") pod \"model-serving-api-75bd89fdfc-xjx5n\" (UID: \"41e62745-afd6-45e5-b3fe-4b9c82669d5a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.446620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.446587 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v92f\" (UniqueName: \"kubernetes.io/projected/41e62745-afd6-45e5-b3fe-4b9c82669d5a-kube-api-access-4v92f\") pod \"model-serving-api-75bd89fdfc-xjx5n\" (UID: \"41e62745-afd6-45e5-b3fe-4b9c82669d5a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.458413 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.458391 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:32.714715 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.714688 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2"] Nov 03 20:37:32.722049 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:37:32.722027 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod09750f2d_31c5_40e7_bd26_deb825796324.slice/crio-8b2fe32d94c5e9da61b2ad9f9ed73b82aec18a633405870c356d0946fc23aed9 WatchSource:0}: Error finding container 8b2fe32d94c5e9da61b2ad9f9ed73b82aec18a633405870c356d0946fc23aed9: Status 404 returned error can't find the container with id 8b2fe32d94c5e9da61b2ad9f9ed73b82aec18a633405870c356d0946fc23aed9 Nov 03 20:37:32.743195 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.743150 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" event={"ID":"09750f2d-31c5-40e7-bd26-deb825796324","Type":"ContainerStarted","Data":"8b2fe32d94c5e9da61b2ad9f9ed73b82aec18a633405870c356d0946fc23aed9"} Nov 03 20:37:32.827900 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:32.827876 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xjx5n"] Nov 03 20:37:32.835568 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:37:32.835541 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e62745_afd6_45e5_b3fe_4b9c82669d5a.slice/crio-81908485b40488269e314465237a0184e767df6e7192f21f721b805160736e91 WatchSource:0}: Error finding container 81908485b40488269e314465237a0184e767df6e7192f21f721b805160736e91: Status 404 returned error can't find the container with id 81908485b40488269e314465237a0184e767df6e7192f21f721b805160736e91 Nov 03 20:37:33.749801 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:33.749772 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" event={"ID":"41e62745-afd6-45e5-b3fe-4b9c82669d5a","Type":"ContainerStarted","Data":"81908485b40488269e314465237a0184e767df6e7192f21f721b805160736e91"} Nov 03 20:37:35.759432 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:35.759399 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" event={"ID":"41e62745-afd6-45e5-b3fe-4b9c82669d5a","Type":"ContainerStarted","Data":"666266cfb771df5e6f412301c3efadbd9ab6b8abd60056a252450f686d159ffd"} Nov 03 20:37:35.759798 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:35.759535 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:35.868792 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:35.868703 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" podStartSLOduration=1.161814204 podStartE2EDuration="3.868688137s" totalImagesPullingTime="2.706873933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:35.867241182 +0000 UTC m=+523.104219054" watchObservedRunningTime="2026-11-03 20:37:35.868688137 +0000 UTC m=+523.105666012" Nov 03 20:37:37.772949 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:37.772907 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" event={"ID":"09750f2d-31c5-40e7-bd26-deb825796324","Type":"ContainerStarted","Data":"4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8"} Nov 03 20:37:37.773283 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:37.773060 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:37:37.881147 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:37.881105 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" podStartSLOduration=0.922377819 podStartE2EDuration="5.881093949s" totalImagesPullingTime="4.95871613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:37.879611568 +0000 UTC m=+525.116589443" watchObservedRunningTime="2026-11-03 20:37:37.881093949 +0000 UTC m=+525.118071822" Nov 03 20:37:46.766751 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:37:46.766722 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-xjx5n" Nov 03 20:37:56.081429 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:37:56.081391 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:38:08.783019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:08.782990 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:38:25.939358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:25.939326 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz"] Nov 03 20:38:25.943122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:25.943104 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.038786 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.038757 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:38:26.038916 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.038825 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nqbhl\"" Nov 03 20:38:26.038916 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.038831 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:38:26.038916 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.038831 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:38:26.039069 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.039002 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:38:26.040313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.040292 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz"] Nov 03 20:38:26.070073 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.070042 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-maas-api-tls\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.070200 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.070139 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-metrics-certs\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.070252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.070204 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.070252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.070237 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h5fj\" (UniqueName: \"kubernetes.io/projected/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-kube-api-access-9h5fj\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.080577 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:38:26.080489 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:38:26.170967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.170941 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-metrics-certs\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.171122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.170981 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.171122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.171018 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h5fj\" (UniqueName: \"kubernetes.io/projected/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-kube-api-access-9h5fj\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.171302 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.171277 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-maas-api-tls\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.171675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.171658 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.173301 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.173279 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-metrics-certs\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.173434 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.173421 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-maas-api-tls\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.223689 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.223636 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h5fj\" (UniqueName: \"kubernetes.io/projected/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-kube-api-access-9h5fj\") pod \"maas-api-6d669c7fcd-t6kgz\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.254610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.254585 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:26.618079 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.618005 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz"] Nov 03 20:38:26.625256 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:38:26.625227 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5815bfc0_e9d0_4e93_b0d8_eef231b7396e.slice/crio-8912b156c3c593cc85904be24fd1974f373df400e58f39f57f22ed888d2c24c8 WatchSource:0}: Error finding container 8912b156c3c593cc85904be24fd1974f373df400e58f39f57f22ed888d2c24c8: Status 404 returned error can't find the container with id 8912b156c3c593cc85904be24fd1974f373df400e58f39f57f22ed888d2c24c8 Nov 03 20:38:26.971445 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:26.971410 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" event={"ID":"5815bfc0-e9d0-4e93-b0d8-eef231b7396e","Type":"ContainerStarted","Data":"8912b156c3c593cc85904be24fd1974f373df400e58f39f57f22ed888d2c24c8"} Nov 03 20:38:29.989624 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:29.989575 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" event={"ID":"5815bfc0-e9d0-4e93-b0d8-eef231b7396e","Type":"ContainerStarted","Data":"3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c"} Nov 03 20:38:29.989988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:29.989714 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:30.098464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:30.098413 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" podStartSLOduration=2.174223657 podStartE2EDuration="5.09839661s" totalImagesPullingTime="2.924172953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:30.097232322 +0000 UTC m=+577.334210195" watchObservedRunningTime="2026-11-03 20:38:30.09839661 +0000 UTC m=+577.335374482" Nov 03 20:38:35.999558 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:35.999488 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:41.342798 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.342758 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc"] Nov 03 20:38:41.346431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.346413 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:41.398956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.398934 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc"] Nov 03 20:38:41.476892 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.476867 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chx66\" (UniqueName: \"kubernetes.io/projected/71213df2-9380-4e6b-9971-73d2404030ae-kube-api-access-chx66\") pod \"llmisvc-controller-manager-6466c554d8-l2jnc\" (UID: \"71213df2-9380-4e6b-9971-73d2404030ae\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:41.477015 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.476918 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71213df2-9380-4e6b-9971-73d2404030ae-cert\") pod \"llmisvc-controller-manager-6466c554d8-l2jnc\" (UID: \"71213df2-9380-4e6b-9971-73d2404030ae\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:41.577672 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.577647 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chx66\" (UniqueName: \"kubernetes.io/projected/71213df2-9380-4e6b-9971-73d2404030ae-kube-api-access-chx66\") pod \"llmisvc-controller-manager-6466c554d8-l2jnc\" (UID: \"71213df2-9380-4e6b-9971-73d2404030ae\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:41.577769 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.577690 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71213df2-9380-4e6b-9971-73d2404030ae-cert\") pod \"llmisvc-controller-manager-6466c554d8-l2jnc\" (UID: \"71213df2-9380-4e6b-9971-73d2404030ae\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:41.579892 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.579874 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71213df2-9380-4e6b-9971-73d2404030ae-cert\") pod \"llmisvc-controller-manager-6466c554d8-l2jnc\" (UID: \"71213df2-9380-4e6b-9971-73d2404030ae\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:41.630459 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.630401 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chx66\" (UniqueName: \"kubernetes.io/projected/71213df2-9380-4e6b-9971-73d2404030ae-kube-api-access-chx66\") pod \"llmisvc-controller-manager-6466c554d8-l2jnc\" (UID: \"71213df2-9380-4e6b-9971-73d2404030ae\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:41.658584 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:41.658558 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:42.027724 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:42.027694 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc"] Nov 03 20:38:42.035927 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:38:42.035900 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71213df2_9380_4e6b_9971_73d2404030ae.slice/crio-72bea36f71f8e8f3db6159dc5f65c3d820fe419c2eead3706270f6cc8ddf39c1 WatchSource:0}: Error finding container 72bea36f71f8e8f3db6159dc5f65c3d820fe419c2eead3706270f6cc8ddf39c1: Status 404 returned error can't find the container with id 72bea36f71f8e8f3db6159dc5f65c3d820fe419c2eead3706270f6cc8ddf39c1 Nov 03 20:38:43.040219 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:43.040184 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" event={"ID":"71213df2-9380-4e6b-9971-73d2404030ae","Type":"ContainerStarted","Data":"b7985a421d7deaa0f4f31352e6929a781d7f93ad4a1e4762a778b5df4d5c8d3a"} Nov 03 20:38:43.040612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:43.040237 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" event={"ID":"71213df2-9380-4e6b-9971-73d2404030ae","Type":"ContainerStarted","Data":"72bea36f71f8e8f3db6159dc5f65c3d820fe419c2eead3706270f6cc8ddf39c1"} Nov 03 20:38:43.040612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:43.040265 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:38:43.148036 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:43.147949 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" podStartSLOduration=1.344498043 podStartE2EDuration="2.14793668s" totalImagesPullingTime="803.438637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:43.146207147 +0000 UTC m=+590.383185018" watchObservedRunningTime="2026-11-03 20:38:43.14793668 +0000 UTC m=+590.384914553" Nov 03 20:38:47.000804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.000763 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz"] Nov 03 20:38:47.001415 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.000946 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" podUID="5815bfc0-e9d0-4e93-b0d8-eef231b7396e" containerName="maas-api" containerID="cri-o://3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c" gracePeriod=30 Nov 03 20:38:47.236171 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.236134 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:47.315874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.315825 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-openshift-service-ca\") pod \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " Nov 03 20:38:47.315874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.315858 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-maas-api-tls\") pod \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " Nov 03 20:38:47.316041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.315892 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-kube-api-access-9h5fj\" (UniqueName: \"kubernetes.io/projected/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-kube-api-access-9h5fj\") pod \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " Nov 03 20:38:47.316041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.315933 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-metrics-certs\") pod \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\" (UID: \"5815bfc0-e9d0-4e93-b0d8-eef231b7396e\") " Nov 03 20:38:47.316193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.316146 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-openshift-service-ca" pod "5815bfc0-e9d0-4e93-b0d8-eef231b7396e" (UID: "5815bfc0-e9d0-4e93-b0d8-eef231b7396e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:47.317954 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.317930 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-maas-api-tls" pod "5815bfc0-e9d0-4e93-b0d8-eef231b7396e" (UID: "5815bfc0-e9d0-4e93-b0d8-eef231b7396e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:47.318065 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.318045 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-kube-api-access-9h5fj" pod "5815bfc0-e9d0-4e93-b0d8-eef231b7396e" (UID: "5815bfc0-e9d0-4e93-b0d8-eef231b7396e"). InnerVolumeSpecName "kube-api-access-9h5fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:47.318105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.318042 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-metrics-certs" pod "5815bfc0-e9d0-4e93-b0d8-eef231b7396e" (UID: "5815bfc0-e9d0-4e93-b0d8-eef231b7396e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:47.417467 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.417439 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h5fj\" (UniqueName: \"kubernetes.io/projected/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-kube-api-access-9h5fj\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:47.417467 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.417466 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:47.417649 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.417476 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:47.417649 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:47.417485 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5815bfc0-e9d0-4e93-b0d8-eef231b7396e-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:48.067794 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.067754 4227 generic.go:356] "Generic (PLEG): container finished" podID="5815bfc0-e9d0-4e93-b0d8-eef231b7396e" containerID="3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c" exitCode=0 Nov 03 20:38:48.068265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.067797 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" Nov 03 20:38:48.068265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.067844 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" event={"ID":"5815bfc0-e9d0-4e93-b0d8-eef231b7396e","Type":"ContainerDied","Data":"3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c"} Nov 03 20:38:48.068265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.067904 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz" event={"ID":"5815bfc0-e9d0-4e93-b0d8-eef231b7396e","Type":"ContainerDied","Data":"8912b156c3c593cc85904be24fd1974f373df400e58f39f57f22ed888d2c24c8"} Nov 03 20:38:48.068265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.067937 4227 scope.go:122] "RemoveContainer" containerID="3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c" Nov 03 20:38:48.076529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.076513 4227 scope.go:122] "RemoveContainer" containerID="3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c" Nov 03 20:38:48.076795 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:38:48.076778 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c\": container with ID starting with 3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c not found: ID does not exist" containerID="3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c" Nov 03 20:38:48.076849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.076803 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c"} err="failed to get container status \"3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c\": rpc error: code = NotFound desc = could not find container \"3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c\": container with ID starting with 3d9154c43044d82fabc4a20b98111470126ea7173354d56abd24c82d58035f6c not found: ID does not exist" Nov 03 20:38:48.218477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.218448 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz"] Nov 03 20:38:48.264973 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.264944 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t6kgz"] Nov 03 20:38:48.897423 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:48.897392 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5815bfc0-e9d0-4e93-b0d8-eef231b7396e" path="/var/lib/kubelet/pods/5815bfc0-e9d0-4e93-b0d8-eef231b7396e/volumes" Nov 03 20:38:52.935716 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:52.935689 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:38:52.936097 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:38:52.935689 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:38:56.079721 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:38:56.079687 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:39:14.052769 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.052735 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l2jnc" Nov 03 20:39:14.213798 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.213766 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2"] Nov 03 20:39:14.214053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.214016 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" podUID="09750f2d-31c5-40e7-bd26-deb825796324" containerName="manager" containerID="cri-o://4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8" gracePeriod=30 Nov 03 20:39:14.455309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.455284 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:39:14.610666 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.610601 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09750f2d-31c5-40e7-bd26-deb825796324-kube-api-access-2fqwn\" (UniqueName: \"kubernetes.io/projected/09750f2d-31c5-40e7-bd26-deb825796324-kube-api-access-2fqwn\") pod \"09750f2d-31c5-40e7-bd26-deb825796324\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " Nov 03 20:39:14.610666 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.610656 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09750f2d-31c5-40e7-bd26-deb825796324-cert\" (UniqueName: \"kubernetes.io/secret/09750f2d-31c5-40e7-bd26-deb825796324-cert\") pod \"09750f2d-31c5-40e7-bd26-deb825796324\" (UID: \"09750f2d-31c5-40e7-bd26-deb825796324\") " Nov 03 20:39:14.612766 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.612737 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09750f2d-31c5-40e7-bd26-deb825796324-cert" pod "09750f2d-31c5-40e7-bd26-deb825796324" (UID: "09750f2d-31c5-40e7-bd26-deb825796324"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:14.612870 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.612803 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09750f2d-31c5-40e7-bd26-deb825796324-kube-api-access-2fqwn" pod "09750f2d-31c5-40e7-bd26-deb825796324" (UID: "09750f2d-31c5-40e7-bd26-deb825796324"). InnerVolumeSpecName "kube-api-access-2fqwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:14.712019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.711993 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fqwn\" (UniqueName: \"kubernetes.io/projected/09750f2d-31c5-40e7-bd26-deb825796324-kube-api-access-2fqwn\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:14.712019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:14.712020 4227 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09750f2d-31c5-40e7-bd26-deb825796324-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:15.173217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.173117 4227 generic.go:356] "Generic (PLEG): container finished" podID="09750f2d-31c5-40e7-bd26-deb825796324" containerID="4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8" exitCode=0 Nov 03 20:39:15.173217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.173183 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" Nov 03 20:39:15.173217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.173198 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" event={"ID":"09750f2d-31c5-40e7-bd26-deb825796324","Type":"ContainerDied","Data":"4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8"} Nov 03 20:39:15.173708 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.173261 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2" event={"ID":"09750f2d-31c5-40e7-bd26-deb825796324","Type":"ContainerDied","Data":"8b2fe32d94c5e9da61b2ad9f9ed73b82aec18a633405870c356d0946fc23aed9"} Nov 03 20:39:15.173708 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.173285 4227 scope.go:122] "RemoveContainer" containerID="4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8" Nov 03 20:39:15.181089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.181070 4227 scope.go:122] "RemoveContainer" containerID="4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8" Nov 03 20:39:15.181359 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:39:15.181339 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8\": container with ID starting with 4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8 not found: ID does not exist" containerID="4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8" Nov 03 20:39:15.181416 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.181368 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8"} err="failed to get container status \"4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8\": rpc error: code = NotFound desc = could not find container \"4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8\": container with ID starting with 4213f3d77a0d2afff91b93e3b16453dc8ceb5cb8104cb4086350641b41c4b4b8 not found: ID does not exist" Nov 03 20:39:15.330976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.330953 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2"] Nov 03 20:39:15.377683 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:15.377658 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4hvg2"] Nov 03 20:39:16.898212 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:39:16.898181 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09750f2d-31c5-40e7-bd26-deb825796324" path="/var/lib/kubelet/pods/09750f2d-31c5-40e7-bd26-deb825796324/volumes" Nov 03 20:39:26.082072 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:39:26.082034 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:39:56.086418 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:39:56.086380 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:40:17.964886 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.964803 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6"] Nov 03 20:40:17.965484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.965150 4227 cpu_manager.go:414] "containerMap: removing container" podUID="09750f2d-31c5-40e7-bd26-deb825796324" containerName="manager" Nov 03 20:40:17.965484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.965174 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09750f2d-31c5-40e7-bd26-deb825796324" containerName="manager" Nov 03 20:40:17.965484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.965186 4227 cpu_manager.go:414] "containerMap: removing container" podUID="5815bfc0-e9d0-4e93-b0d8-eef231b7396e" containerName="maas-api" Nov 03 20:40:17.965484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.965191 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5815bfc0-e9d0-4e93-b0d8-eef231b7396e" containerName="maas-api" Nov 03 20:40:17.965484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.965248 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="09750f2d-31c5-40e7-bd26-deb825796324" containerName="manager" Nov 03 20:40:17.965484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.965258 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="5815bfc0-e9d0-4e93-b0d8-eef231b7396e" containerName="maas-api" Nov 03 20:40:17.968261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:17.968242 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.016958 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.016924 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.016973 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017026 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017049 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghpht\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-kube-api-access-ghpht\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017199 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017242 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017272 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017282 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spnvt\"" Nov 03 20:40:18.017466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017310 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718f11f-7b9d-4097-8d93-98c9195ab382-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.017466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.017406 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.065077 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.065045 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6"] Nov 03 20:40:18.117740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117713 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.117898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117745 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghpht\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-kube-api-access-ghpht\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.117898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117767 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.117898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117791 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.117898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117809 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.117898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117833 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718f11f-7b9d-4097-8d93-98c9195ab382-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.117898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117860 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.117898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117894 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.118294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.117920 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.118294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.118202 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.118546 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.118513 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.118657 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.118576 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.118714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.118670 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.118714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.118685 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718f11f-7b9d-4097-8d93-98c9195ab382-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.120666 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.120641 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.120794 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.120778 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.170872 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.170846 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.171579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.171552 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghpht\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-kube-api-access-ghpht\") pod \"maas-default-gateway-openshift-default-696f47c6c-h62q6\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.279881 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.279814 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:18.705314 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.705282 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6"] Nov 03 20:40:18.750202 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:40:18.750151 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8718f11f_7b9d_4097_8d93_98c9195ab382.slice/crio-b9aea9d7ef9efca60aaef973f0726ba387c0a5fcf53a528f3175f19fb05794d3 WatchSource:0}: Error finding container b9aea9d7ef9efca60aaef973f0726ba387c0a5fcf53a528f3175f19fb05794d3: Status 404 returned error can't find the container with id b9aea9d7ef9efca60aaef973f0726ba387c0a5fcf53a528f3175f19fb05794d3 Nov 03 20:40:18.752169 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:18.752141 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:40:19.429348 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:19.429312 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" event={"ID":"8718f11f-7b9d-4097-8d93-98c9195ab382","Type":"ContainerStarted","Data":"b9aea9d7ef9efca60aaef973f0726ba387c0a5fcf53a528f3175f19fb05794d3"} Nov 03 20:40:22.590387 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:22.590352 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:40:22.590683 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:22.590420 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:40:22.590683 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:22.590474 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:40:23.462183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:23.459337 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" event={"ID":"8718f11f-7b9d-4097-8d93-98c9195ab382","Type":"ContainerStarted","Data":"c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3"} Nov 03 20:40:23.665240 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:23.665172 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podStartSLOduration=2.827955273 podStartE2EDuration="6.66513687s" totalImagesPullingTime="3.837181597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:23.617795475 +0000 UTC m=+690.854773349" watchObservedRunningTime="2026-11-03 20:40:23.66513687 +0000 UTC m=+690.902114744" Nov 03 20:40:24.280499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:24.280459 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:24.282073 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:24.282038 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-h62q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 20:40:24.282220 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:24.282100 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 20:40:25.280738 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:25.280701 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-h62q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 20:40:25.281183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:25.280758 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 20:40:26.095497 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:40:26.095464 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:40:26.280712 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:26.280678 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-h62q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 20:40:26.280883 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:26.280732 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 20:40:27.280630 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:27.280596 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-h62q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 20:40:27.280800 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:27.280649 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 20:40:28.280423 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:28.280391 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-h62q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 20:40:28.280825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:28.280442 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 20:40:29.280515 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:29.280480 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-h62q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 20:40:29.280992 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:29.280534 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 20:40:30.794622 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:30.794591 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:30.795046 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:30.794939 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:30.795830 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:30.795807 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:40:33.095537 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.095493 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5"] Nov 03 20:40:33.098921 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.098904 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141347 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141379 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dngm\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-kube-api-access-4dngm\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141589 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141414 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141589 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141535 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141589 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141568 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141748 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141632 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b1294-920b-4389-a804-2a31f2418af9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141748 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141659 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141748 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141703 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.141748 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.141736 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b1294-920b-4389-a804-2a31f2418af9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.195604 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.195571 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5"] Nov 03 20:40:33.242112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242067 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242123 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dngm\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-kube-api-access-4dngm\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242170 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242211 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242240 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242443 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242290 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b1294-920b-4389-a804-2a31f2418af9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242443 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242327 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242443 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242353 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242443 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242383 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b1294-920b-4389-a804-2a31f2418af9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242730 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242708 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242819 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242730 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.242949 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242924 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.243031 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.242954 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.243171 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.243141 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b1294-920b-4389-a804-2a31f2418af9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.244404 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.244385 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.244656 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.244638 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b1294-920b-4389-a804-2a31f2418af9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.294835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.294812 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dngm\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-kube-api-access-4dngm\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.294986 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.294969 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-p64p5\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.413797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.413766 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:33.832431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.832363 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5"] Nov 03 20:40:33.839986 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:40:33.839960 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239b1294_920b_4389_a804_2a31f2418af9.slice/crio-a213fd8061dce73f88e1f09fa6ada54516a4fafccdc9beb8a826936d57bd0168 WatchSource:0}: Error finding container a213fd8061dce73f88e1f09fa6ada54516a4fafccdc9beb8a826936d57bd0168: Status 404 returned error can't find the container with id a213fd8061dce73f88e1f09fa6ada54516a4fafccdc9beb8a826936d57bd0168 Nov 03 20:40:33.842003 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.841968 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:40:33.842104 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.842036 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:40:33.842104 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:33.842069 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:40:34.517784 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:34.517749 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" event={"ID":"239b1294-920b-4389-a804-2a31f2418af9","Type":"ContainerStarted","Data":"4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da"} Nov 03 20:40:34.518241 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:34.517798 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" event={"ID":"239b1294-920b-4389-a804-2a31f2418af9","Type":"ContainerStarted","Data":"a213fd8061dce73f88e1f09fa6ada54516a4fafccdc9beb8a826936d57bd0168"} Nov 03 20:40:34.724848 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:34.724786 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podStartSLOduration=1.724769023 podStartE2EDuration="1.724769023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:34.677609793 +0000 UTC m=+701.914587667" watchObservedRunningTime="2026-11-03 20:40:34.724769023 +0000 UTC m=+701.961746901" Nov 03 20:40:35.414175 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:35.414134 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:35.415813 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:35.415779 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p64p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 20:40:35.415944 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:35.415827 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 20:40:36.415152 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:36.415113 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p64p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 20:40:36.415557 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:36.415195 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 20:40:37.414787 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:37.414747 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p64p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 20:40:37.414958 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:37.414806 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 20:40:38.414816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:38.414783 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p64p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 20:40:38.415208 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:38.414838 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 20:40:39.414888 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:39.414849 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p64p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 20:40:39.415305 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:39.414903 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 20:40:40.414225 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:40.414190 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p64p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 20:40:40.414376 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:40.414245 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 20:40:41.951053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:41.951018 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:41.951555 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:41.951436 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:41.952128 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:40:41.952109 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:40:56.079892 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:40:56.079857 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:41:18.419513 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.419479 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt"] Nov 03 20:41:18.422864 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.422839 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.519752 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.519724 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt"] Nov 03 20:41:18.535752 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.535723 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.535859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.535757 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.535859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.535780 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.535859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.535799 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d396c328-e385-41d3-a1a0-ab54183aff2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.535966 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.535903 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.535966 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.535948 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnchv\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-kube-api-access-dnchv\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.536027 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.536010 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.536062 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.536041 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.536094 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.536083 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637254 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637228 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637269 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637288 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637313 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637341 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637375 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637400 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d396c328-e385-41d3-a1a0-ab54183aff2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637451 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637501 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnchv\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-kube-api-access-dnchv\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637741 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637718 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.637804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.637755 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.638104 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.638080 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.638220 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.638132 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d396c328-e385-41d3-a1a0-ab54183aff2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.638220 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.638130 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.639695 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.639666 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.639811 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.639795 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.690506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.690457 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.690714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.690697 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnchv\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-kube-api-access-dnchv\") pod \"maas-default-gateway-openshift-default-696f47c6c-lpszt\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:18.735814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:18.735795 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:19.152854 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:19.152824 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt"] Nov 03 20:41:19.159814 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:41:19.159789 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd396c328_e385_41d3_a1a0_ab54183aff2b.slice/crio-435307ac5fc3ec8c9c90c410a80784b16e0d641d3da856b77323039d66db07ee WatchSource:0}: Error finding container 435307ac5fc3ec8c9c90c410a80784b16e0d641d3da856b77323039d66db07ee: Status 404 returned error can't find the container with id 435307ac5fc3ec8c9c90c410a80784b16e0d641d3da856b77323039d66db07ee Nov 03 20:41:19.161838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:19.161806 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:41:19.161928 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:19.161884 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:41:19.161987 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:19.161927 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:41:19.708975 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:19.708938 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" event={"ID":"d396c328-e385-41d3-a1a0-ab54183aff2b","Type":"ContainerStarted","Data":"c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d"} Nov 03 20:41:19.709401 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:19.708991 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" event={"ID":"d396c328-e385-41d3-a1a0-ab54183aff2b","Type":"ContainerStarted","Data":"435307ac5fc3ec8c9c90c410a80784b16e0d641d3da856b77323039d66db07ee"} Nov 03 20:41:19.736836 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:19.736811 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:20.737731 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:20.737697 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-lpszt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:41:20.738122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:20.737747 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 20:41:21.736363 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:21.736325 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-lpszt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:41:21.736559 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:21.736409 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 20:41:22.736334 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:22.736301 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-lpszt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:41:22.736723 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:22.736366 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 20:41:23.736221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:23.736189 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-lpszt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:41:23.736414 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:23.736247 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 20:41:24.736689 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:24.736656 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-lpszt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:41:24.737146 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:24.736722 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 20:41:25.736792 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:25.736760 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-lpszt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:41:25.737193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:25.736812 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 03 20:41:26.080703 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:41:26.080624 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:41:27.200673 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:27.200630 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:27.201259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:27.201043 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:27.201991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:27.201968 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 20:41:27.403390 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:41:27.403349 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podStartSLOduration=9.403337995 podStartE2EDuration="9.403337995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:19.823003154 +0000 UTC m=+747.059981026" watchObservedRunningTime="2026-11-03 20:41:27.403337995 +0000 UTC m=+754.640315867" Nov 03 20:41:56.080035 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:41:56.079992 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:42:08.202892 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.202860 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr"] Nov 03 20:42:08.206710 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.206687 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.256623 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.256594 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-41aa21-openshift-default-dockercfg-plbb2\"" Nov 03 20:42:08.305607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.305572 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr"] Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348270 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348378 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348476 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348594 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348766 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6d7\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-kube-api-access-rc6d7\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348856 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d178850-07af-4e82-858d-b9e4e181b8c5-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348919 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348950 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.355180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.348992 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.449848 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.449814 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450097 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.449884 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450410 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450385 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450511 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450471 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6d7\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-kube-api-access-rc6d7\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450573 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450517 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d178850-07af-4e82-858d-b9e4e181b8c5-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450668 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450644 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450730 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450671 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450730 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450722 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450836 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450762 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450824 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.450890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.450861 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.451039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.451010 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.451617 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.451586 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d178850-07af-4e82-858d-b9e4e181b8c5-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.451790 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.451765 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.454225 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.454001 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.454988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.454949 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.504731 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.504692 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.504875 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.504832 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6d7\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-kube-api-access-rc6d7\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.520501 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.520447 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:08.945755 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.945719 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr"] Nov 03 20:42:08.952491 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:42:08.952446 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d178850_07af_4e82_858d_b9e4e181b8c5.slice/crio-851a777f77c67e522de3f2f5b9d3e5c7f5f58682478f7cb04b123abf8d934665 WatchSource:0}: Error finding container 851a777f77c67e522de3f2f5b9d3e5c7f5f58682478f7cb04b123abf8d934665: Status 404 returned error can't find the container with id 851a777f77c67e522de3f2f5b9d3e5c7f5f58682478f7cb04b123abf8d934665 Nov 03 20:42:08.954664 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.954636 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:42:08.954735 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.954698 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:42:08.954735 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:08.954727 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:42:09.913624 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:09.913590 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" event={"ID":"2d178850-07af-4e82-858d-b9e4e181b8c5","Type":"ContainerStarted","Data":"0a470c8590929436ff0ce92faecedbb06abaa4ebce7791ea4eb42156c85750f4"} Nov 03 20:42:09.914014 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:09.913638 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" event={"ID":"2d178850-07af-4e82-858d-b9e4e181b8c5","Type":"ContainerStarted","Data":"851a777f77c67e522de3f2f5b9d3e5c7f5f58682478f7cb04b123abf8d934665"} Nov 03 20:42:10.071098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:10.071050 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podStartSLOduration=2.071037371 podStartE2EDuration="2.071037371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:10.069591028 +0000 UTC m=+797.306568903" watchObservedRunningTime="2026-11-03 20:42:10.071037371 +0000 UTC m=+797.308015243" Nov 03 20:42:10.520806 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:10.520773 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:10.525713 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:10.525688 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:10.917458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:10.917431 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:10.918112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:10.918095 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:17.215973 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.215939 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28"] Nov 03 20:42:17.219476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.219457 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.229378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.229354 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.229464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.229395 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00237c14-2238-4ed7-b408-3984cc0dfece-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.229504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.229463 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fg4h\" (UniqueName: \"kubernetes.io/projected/00237c14-2238-4ed7-b408-3984cc0dfece-kube-api-access-4fg4h\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.229504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.229490 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-metrics-certs\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.315129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.315102 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-fd9afe-serving-cert\"" Nov 03 20:42:17.315129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.315105 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:42:17.315344 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.315145 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nqbhl\"" Nov 03 20:42:17.315344 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.315212 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:42:17.315344 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.315236 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:42:17.316938 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.316920 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28"] Nov 03 20:42:17.329854 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.329832 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.329906 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.329866 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00237c14-2238-4ed7-b408-3984cc0dfece-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.329906 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.329887 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fg4h\" (UniqueName: \"kubernetes.io/projected/00237c14-2238-4ed7-b408-3984cc0dfece-kube-api-access-4fg4h\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.329967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.329909 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-metrics-certs\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.329998 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:17.329961 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-serving-cert: secret "maas-api-e2e-worker-w0-fd9afe-serving-cert" not found Nov 03 20:42:17.330043 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:17.330033 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls podName:00237c14-2238-4ed7-b408-3984cc0dfece nodeName:}" failed. No retries permitted until 2026-11-03 20:42:17.830016412 +0000 UTC m=+805.066994277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls") pod "maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" (UID: "00237c14-2238-4ed7-b408-3984cc0dfece") : secret "maas-api-e2e-worker-w0-fd9afe-serving-cert" not found Nov 03 20:42:17.330490 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.330466 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00237c14-2238-4ed7-b408-3984cc0dfece-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.332241 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.332221 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-metrics-certs\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.383869 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.383841 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fg4h\" (UniqueName: \"kubernetes.io/projected/00237c14-2238-4ed7-b408-3984cc0dfece-kube-api-access-4fg4h\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.835265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.835232 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:17.837545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:17.837526 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:18.132509 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:18.132415 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:18.498122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:18.498083 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28"] Nov 03 20:42:18.506109 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:42:18.506083 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00237c14_2238_4ed7_b408_3984cc0dfece.slice/crio-3ff1cae18faee4020fda2bd88aec9fb7d934d620540e6b6661f22666593df074 WatchSource:0}: Error finding container 3ff1cae18faee4020fda2bd88aec9fb7d934d620540e6b6661f22666593df074: Status 404 returned error can't find the container with id 3ff1cae18faee4020fda2bd88aec9fb7d934d620540e6b6661f22666593df074 Nov 03 20:42:18.967788 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:18.967753 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" event={"ID":"00237c14-2238-4ed7-b408-3984cc0dfece","Type":"ContainerStarted","Data":"3ff1cae18faee4020fda2bd88aec9fb7d934d620540e6b6661f22666593df074"} Nov 03 20:42:19.973385 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:19.973341 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" event={"ID":"00237c14-2238-4ed7-b408-3984cc0dfece","Type":"ContainerStarted","Data":"8cb6310ec58a4d3aa7387d3131fdffd6d26a74d48e4ef0c5653450780a6f9cea"} Nov 03 20:42:19.973791 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:19.973502 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:20.082818 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:20.082765 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" podStartSLOduration=2.182234861 podStartE2EDuration="3.082751877s" totalImagesPullingTime="900.517016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:20.081074201 +0000 UTC m=+807.318052075" watchObservedRunningTime="2026-11-03 20:42:20.082751877 +0000 UTC m=+807.319729749" Nov 03 20:42:25.996120 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:25.996092 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 20:42:26.096916 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:26.096817 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:42:27.709010 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.708966 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk"] Nov 03 20:42:27.712552 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.712535 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.742178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.742140 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2pc\" (UniqueName: \"kubernetes.io/projected/2f7bdc91-c5c7-4abd-9993-815c3623b258-kube-api-access-xk2pc\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.742273 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.742204 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-metrics-certs\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.742313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.742281 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.742313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.742306 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7bdc91-c5c7-4abd-9993-815c3623b258-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.809272 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.809249 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-1ec892-serving-cert\"" Nov 03 20:42:27.810807 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.810788 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk"] Nov 03 20:42:27.843097 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.843076 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7bdc91-c5c7-4abd-9993-815c3623b258-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.843234 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.843129 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2pc\" (UniqueName: \"kubernetes.io/projected/2f7bdc91-c5c7-4abd-9993-815c3623b258-kube-api-access-xk2pc\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.843234 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.843193 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-metrics-certs\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.843234 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.843212 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.843399 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:27.843299 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-serving-cert: secret "maas-api-e2e-worker-w5-1ec892-serving-cert" not found Nov 03 20:42:27.843399 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:27.843353 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls podName:2f7bdc91-c5c7-4abd-9993-815c3623b258 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:28.343334613 +0000 UTC m=+815.580312478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls") pod "maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" (UID: "2f7bdc91-c5c7-4abd-9993-815c3623b258") : secret "maas-api-e2e-worker-w5-1ec892-serving-cert" not found Nov 03 20:42:27.843815 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.843799 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7bdc91-c5c7-4abd-9993-815c3623b258-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.845504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.845484 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-metrics-certs\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:27.899694 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:27.899667 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2pc\" (UniqueName: \"kubernetes.io/projected/2f7bdc91-c5c7-4abd-9993-815c3623b258-kube-api-access-xk2pc\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:28.348371 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:28.348330 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:28.350673 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:28.350643 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:28.624385 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:28.624307 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:28.990761 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:28.990726 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk"] Nov 03 20:42:28.997914 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:42:28.997768 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7bdc91_c5c7_4abd_9993_815c3623b258.slice/crio-18460c20b6f5d1b6511ac2e84a89ffce0ea2ad6981ac7af0edf6e7bbdfe837f1 WatchSource:0}: Error finding container 18460c20b6f5d1b6511ac2e84a89ffce0ea2ad6981ac7af0edf6e7bbdfe837f1: Status 404 returned error can't find the container with id 18460c20b6f5d1b6511ac2e84a89ffce0ea2ad6981ac7af0edf6e7bbdfe837f1 Nov 03 20:42:29.034071 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:29.034045 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" event={"ID":"2f7bdc91-c5c7-4abd-9993-815c3623b258","Type":"ContainerStarted","Data":"18460c20b6f5d1b6511ac2e84a89ffce0ea2ad6981ac7af0edf6e7bbdfe837f1"} Nov 03 20:42:30.041211 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:30.041173 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" event={"ID":"2f7bdc91-c5c7-4abd-9993-815c3623b258","Type":"ContainerStarted","Data":"77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0"} Nov 03 20:42:30.041661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:30.041296 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:30.148841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:30.148786 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" podStartSLOduration=2.355910148 podStartE2EDuration="3.148771145s" totalImagesPullingTime="792.860997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:30.14721575 +0000 UTC m=+817.384193624" watchObservedRunningTime="2026-11-03 20:42:30.148771145 +0000 UTC m=+817.385749059" Nov 03 20:42:36.075247 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:36.075218 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 20:42:38.377544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.377495 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd"] Nov 03 20:42:38.380824 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.380802 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.436120 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.436085 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-b119f2-serving-cert\"" Nov 03 20:42:38.470710 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.470663 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29299fe-e752-4a30-bcad-a827cac73147-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.470975 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.470954 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.471153 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.471133 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-metrics-certs\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.475412 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.475387 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxv2c\" (UniqueName: \"kubernetes.io/projected/f29299fe-e752-4a30-bcad-a827cac73147-kube-api-access-gxv2c\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.479960 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.479927 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd"] Nov 03 20:42:38.577644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.577608 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29299fe-e752-4a30-bcad-a827cac73147-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.577644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.577650 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.577926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.577702 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-metrics-certs\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.577926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.577732 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxv2c\" (UniqueName: \"kubernetes.io/projected/f29299fe-e752-4a30-bcad-a827cac73147-kube-api-access-gxv2c\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.578216 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:38.578191 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-serving-cert: secret "maas-api-e2e-worker-w2-b119f2-serving-cert" not found Nov 03 20:42:38.578710 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.578686 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29299fe-e752-4a30-bcad-a827cac73147-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.578855 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:38.578840 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls podName:f29299fe-e752-4a30-bcad-a827cac73147 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:39.078698756 +0000 UTC m=+826.315676609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls") pod "maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" (UID: "f29299fe-e752-4a30-bcad-a827cac73147") : secret "maas-api-e2e-worker-w2-b119f2-serving-cert" not found Nov 03 20:42:38.580923 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.580898 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-metrics-certs\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:38.636805 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:38.634108 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxv2c\" (UniqueName: \"kubernetes.io/projected/f29299fe-e752-4a30-bcad-a827cac73147-kube-api-access-gxv2c\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:39.085304 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:39.085264 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:39.087764 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:39.087732 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:39.293230 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:39.293195 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:39.665676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:39.665639 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd"] Nov 03 20:42:39.674890 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:42:39.674856 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29299fe_e752_4a30_bcad_a827cac73147.slice/crio-b681c2982652efbbee9d3b70cc0de9285ce570c9105672797ed520c44b8b44e3 WatchSource:0}: Error finding container b681c2982652efbbee9d3b70cc0de9285ce570c9105672797ed520c44b8b44e3: Status 404 returned error can't find the container with id b681c2982652efbbee9d3b70cc0de9285ce570c9105672797ed520c44b8b44e3 Nov 03 20:42:40.106293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:40.106197 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" event={"ID":"f29299fe-e752-4a30-bcad-a827cac73147","Type":"ContainerStarted","Data":"b681c2982652efbbee9d3b70cc0de9285ce570c9105672797ed520c44b8b44e3"} Nov 03 20:42:41.140864 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:41.140827 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" event={"ID":"f29299fe-e752-4a30-bcad-a827cac73147","Type":"ContainerStarted","Data":"7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943"} Nov 03 20:42:41.143907 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:41.143875 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:41.251830 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:41.251768 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" podStartSLOduration=2.433766009 podStartE2EDuration="3.251751945s" totalImagesPullingTime="817.985936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:41.250408244 +0000 UTC m=+828.487386244" watchObservedRunningTime="2026-11-03 20:42:41.251751945 +0000 UTC m=+828.488729818" Nov 03 20:42:41.959292 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:41.959256 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:42:41.959461 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:41.959318 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:42:42.146947 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:42.146912 4227 generic.go:356] "Generic (PLEG): container finished" podID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerID="0a470c8590929436ff0ce92faecedbb06abaa4ebce7791ea4eb42156c85750f4" exitCode=0 Nov 03 20:42:42.147458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:42.146988 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" event={"ID":"2d178850-07af-4e82-858d-b9e4e181b8c5","Type":"ContainerDied","Data":"0a470c8590929436ff0ce92faecedbb06abaa4ebce7791ea4eb42156c85750f4"} Nov 03 20:42:42.147458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:42.147434 4227 scope.go:122] "RemoveContainer" containerID="0a470c8590929436ff0ce92faecedbb06abaa4ebce7791ea4eb42156c85750f4" Nov 03 20:42:42.148636 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:42.148607 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:42:42.148718 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:42.148677 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:42:42.148755 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:42.148723 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:42:43.176375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:43.176336 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" event={"ID":"2d178850-07af-4e82-858d-b9e4e181b8c5","Type":"ContainerStarted","Data":"4380218ce1864ba5577416313d5480933dc045baf3c63202f93cba2ee03246cc"} Nov 03 20:42:43.520845 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:43.520762 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:43.520993 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:43.520977 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 20:42:43.521041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:43.521021 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 20:42:44.521415 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:44.521377 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 20:42:44.521915 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:44.521433 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 20:42:45.520949 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:45.520915 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 20:42:45.521120 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:45.520966 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 20:42:46.521635 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:46.521599 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 20:42:46.522025 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:46.521652 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 20:42:47.175032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:47.175002 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 20:42:47.521215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:47.521117 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 20:42:47.521363 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:47.521217 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 20:42:48.524651 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:48.524621 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:48.525106 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:48.524949 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:48.525588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:48.525573 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:42:54.302524 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.302483 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z"] Nov 03 20:42:54.306135 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.306114 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.356886 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.356866 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-984cbc-serving-cert\"" Nov 03 20:42:54.357957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.357931 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z"] Nov 03 20:42:54.456279 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.456240 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l5sd\" (UniqueName: \"kubernetes.io/projected/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-kube-api-access-6l5sd\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.456450 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.456313 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.456450 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.456429 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-metrics-certs\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.456568 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.456543 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.557753 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.557678 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.557753 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.557724 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5sd\" (UniqueName: \"kubernetes.io/projected/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-kube-api-access-6l5sd\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.557930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.557764 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.557930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.557795 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-metrics-certs\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.558455 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.558425 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.560194 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.560147 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-metrics-certs\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.560313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.560293 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.610381 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.610351 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5sd\" (UniqueName: \"kubernetes.io/projected/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-kube-api-access-6l5sd\") pod \"maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:54.618167 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:54.618133 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:55.018567 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:55.018533 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z"] Nov 03 20:42:55.026654 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:42:55.026625 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e0ffb6_e265_40b3_9e31_6dfe02ee743d.slice/crio-3c76dc650a7a555747d1ffc0bd17548f22cf4e009cdea84771cfcba1cfe5737e WatchSource:0}: Error finding container 3c76dc650a7a555747d1ffc0bd17548f22cf4e009cdea84771cfcba1cfe5737e: Status 404 returned error can't find the container with id 3c76dc650a7a555747d1ffc0bd17548f22cf4e009cdea84771cfcba1cfe5737e Nov 03 20:42:55.229956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:55.229918 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" event={"ID":"97e0ffb6-e265-40b3-9e31-6dfe02ee743d","Type":"ContainerStarted","Data":"3c76dc650a7a555747d1ffc0bd17548f22cf4e009cdea84771cfcba1cfe5737e"} Nov 03 20:42:56.080809 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:42:56.080774 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:42:56.238820 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:56.238727 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" event={"ID":"97e0ffb6-e265-40b3-9e31-6dfe02ee743d","Type":"ContainerStarted","Data":"6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529"} Nov 03 20:42:56.239918 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:56.239888 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:42:56.346801 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:42:56.346732 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" podStartSLOduration=1.4479494210000001 podStartE2EDuration="2.346713808s" totalImagesPullingTime="898.764387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:56.346040189 +0000 UTC m=+843.583018062" watchObservedRunningTime="2026-11-03 20:42:56.346713808 +0000 UTC m=+843.583691683" Nov 03 20:43:03.253394 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:03.253361 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 20:43:22.900705 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.900673 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5"] Nov 03 20:43:22.904281 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.904264 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937099 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937139 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937181 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937283 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937486 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937331 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k29dq\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-kube-api-access-k29dq\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937486 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937454 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937599 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937495 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937599 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937578 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b690173-19d4-4bdc-8606-3a144223e616-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.937679 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.937618 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b690173-19d4-4bdc-8606-3a144223e616-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:22.953084 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:22.953054 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fd9afe-openshift-default-dockercfg-k948v\"" Nov 03 20:43:23.001110 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.001080 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5"] Nov 03 20:43:23.038578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038547 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b690173-19d4-4bdc-8606-3a144223e616-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.038749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038608 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.038749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038635 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.038749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038654 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.038749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038673 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.038749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038696 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k29dq\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-kube-api-access-k29dq\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.038749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038729 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.039095 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038759 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.039095 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.038789 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b690173-19d4-4bdc-8606-3a144223e616-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.039246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.039119 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.039246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.039193 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.039352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.039326 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b690173-19d4-4bdc-8606-3a144223e616-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.039466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.039440 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.039529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.039448 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.041047 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.041022 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.041171 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.041139 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b690173-19d4-4bdc-8606-3a144223e616-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.092408 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.092381 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.092830 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.092804 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k29dq\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-kube-api-access-k29dq\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.216656 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.216560 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:23.639263 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.639183 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5"] Nov 03 20:43:23.646458 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:43:23.646430 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b690173_19d4_4bdc_8606_3a144223e616.slice/crio-6542b19c632699a63f664251338938b34726a9bb0ec76bf4c4431316d6aa0c13 WatchSource:0}: Error finding container 6542b19c632699a63f664251338938b34726a9bb0ec76bf4c4431316d6aa0c13: Status 404 returned error can't find the container with id 6542b19c632699a63f664251338938b34726a9bb0ec76bf4c4431316d6aa0c13 Nov 03 20:43:23.648607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.648579 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:43:23.648675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.648642 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:43:23.648675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:23.648671 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:43:24.378530 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:24.378487 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" event={"ID":"4b690173-19d4-4bdc-8606-3a144223e616","Type":"ContainerStarted","Data":"9491ab689dfd4f9de1ac69f8d856f18fbb921601b3f6e2d73e46f35a2aaf222d"} Nov 03 20:43:24.379077 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:24.378551 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" event={"ID":"4b690173-19d4-4bdc-8606-3a144223e616","Type":"ContainerStarted","Data":"6542b19c632699a63f664251338938b34726a9bb0ec76bf4c4431316d6aa0c13"} Nov 03 20:43:24.536211 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:24.536135 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podStartSLOduration=2.536120736 podStartE2EDuration="2.536120736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:24.533979826 +0000 UTC m=+871.770957700" watchObservedRunningTime="2026-11-03 20:43:24.536120736 +0000 UTC m=+871.773098611" Nov 03 20:43:25.217707 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:25.217665 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:25.219139 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:25.219119 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:43:25.219204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:25.219179 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:43:26.104264 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:43:26.104222 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:43:26.217827 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:26.217788 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:43:26.217994 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:26.217856 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:43:27.217069 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:27.217036 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:43:27.217458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:27.217085 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:43:28.217295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:28.217263 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:43:28.217718 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:28.217314 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:43:29.217484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:29.217449 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:43:29.217894 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:29.217499 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:43:30.222427 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:30.222386 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:30.222796 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:30.222765 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:30.223394 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:30.223372 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:43:45.652523 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.652484 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n"] Nov 03 20:43:45.656112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.656092 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.709054 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.709023 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n"] Nov 03 20:43:45.753281 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.753253 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"openshift-service-ca.crt\"" Nov 03 20:43:45.753430 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.753404 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"kube-root-ca.crt\"" Nov 03 20:43:45.753575 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.753558 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"default-dockercfg-596g2\"" Nov 03 20:43:45.753649 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.753632 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"premium-w0-fd9afe-kserve-self-signed-certs\"" Nov 03 20:43:45.791505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.791480 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tls-certs\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.791645 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.791515 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-model-cache\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.791645 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.791547 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-dshm\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.791789 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.791763 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-home\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.791846 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.791805 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwm9\" (UniqueName: \"kubernetes.io/projected/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-kube-api-access-8jwm9\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.791901 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.791866 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tmp-dir\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893261 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tls-certs\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893296 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-model-cache\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893534 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893319 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-dshm\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893534 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893381 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-home\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893534 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893399 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwm9\" (UniqueName: \"kubernetes.io/projected/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-kube-api-access-8jwm9\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893534 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893442 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tmp-dir\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893741 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893675 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-model-cache\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893796 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893776 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-home\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.893832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.893810 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tmp-dir\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.895565 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.895546 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-dshm\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.895754 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.895737 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tls-certs\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.946627 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.946565 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwm9\" (UniqueName: \"kubernetes.io/projected/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-kube-api-access-8jwm9\") pod \"premium-w0-fd9afe-kserve-67dfcd684f-gll8n\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:45.977222 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:45.977191 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:46.376526 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:46.376452 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n"] Nov 03 20:43:46.383589 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:43:46.383552 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb61ce70c_6d1c_41c7_8bc8_988ad23a3d40.slice/crio-2cbcbe761bfda585109ce220c59bc220af4a0b6e50c238e72bfc12cee57aec66 WatchSource:0}: Error finding container 2cbcbe761bfda585109ce220c59bc220af4a0b6e50c238e72bfc12cee57aec66: Status 404 returned error can't find the container with id 2cbcbe761bfda585109ce220c59bc220af4a0b6e50c238e72bfc12cee57aec66 Nov 03 20:43:46.481191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:46.481142 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" event={"ID":"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40","Type":"ContainerStarted","Data":"2cbcbe761bfda585109ce220c59bc220af4a0b6e50c238e72bfc12cee57aec66"} Nov 03 20:43:48.492777 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:48.492744 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" event={"ID":"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40","Type":"ContainerStarted","Data":"a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f"} Nov 03 20:43:48.601690 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:48.601642 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podStartSLOduration=2.01946298 podStartE2EDuration="3.601628368s" totalImagesPullingTime="1.582165388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:48.600102926 +0000 UTC m=+895.837080802" watchObservedRunningTime="2026-11-03 20:43:48.601628368 +0000 UTC m=+895.838606241" Nov 03 20:43:48.978309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:48.978268 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:52.975086 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:52.975060 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:43:52.976328 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:52.976306 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:43:55.977630 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:55.977595 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:55.990090 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:55.990066 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:43:56.082485 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:43:56.082447 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:43:56.555652 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:43:56.555619 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:44:16.913925 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.913886 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5"] Nov 03 20:44:16.918270 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.918248 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:16.987720 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.987690 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-dshm\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:16.987868 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.987741 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68e1972-6268-4597-9520-c1a521a3d998-tls-certs\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:16.987868 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.987821 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-tmp-dir\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:16.987952 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.987903 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-home\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:16.987952 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.987934 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpvh\" (UniqueName: \"kubernetes.io/projected/e68e1972-6268-4597-9520-c1a521a3d998-kube-api-access-cbpvh\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:16.988054 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:16.987977 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-model-cache\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.014493 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.014470 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"openshift-service-ca.crt\"" Nov 03 20:44:17.014628 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.014494 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"kube-root-ca.crt\"" Nov 03 20:44:17.014715 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.014696 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"default-dockercfg-2nc52\"" Nov 03 20:44:17.014768 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.014723 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"premium-w1-41aa21-kserve-self-signed-certs\"" Nov 03 20:44:17.016385 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.016367 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5"] Nov 03 20:44:17.088929 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.088883 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-dshm\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.088979 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68e1972-6268-4597-9520-c1a521a3d998-tls-certs\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.089033 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-tmp-dir\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.089091 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-home\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089319 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.089133 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpvh\" (UniqueName: \"kubernetes.io/projected/e68e1972-6268-4597-9520-c1a521a3d998-kube-api-access-cbpvh\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089319 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.089200 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-model-cache\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089485 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.089465 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-tmp-dir\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.089603 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-model-cache\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.089786 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.089632 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-home\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.091389 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.091367 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-dshm\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.091870 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.091852 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68e1972-6268-4597-9520-c1a521a3d998-tls-certs\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.154030 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.153996 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpvh\" (UniqueName: \"kubernetes.io/projected/e68e1972-6268-4597-9520-c1a521a3d998-kube-api-access-cbpvh\") pod \"premium-w1-41aa21-kserve-9678c9d89-5b9b5\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.231059 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.230976 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:17.628210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.628114 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5"] Nov 03 20:44:17.635044 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:44:17.635022 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode68e1972_6268_4597_9520_c1a521a3d998.slice/crio-f296c78910f445256e50a0fe9b736a3c23134ab4d4c5e7089207174c94c996e7 WatchSource:0}: Error finding container f296c78910f445256e50a0fe9b736a3c23134ab4d4c5e7089207174c94c996e7: Status 404 returned error can't find the container with id f296c78910f445256e50a0fe9b736a3c23134ab4d4c5e7089207174c94c996e7 Nov 03 20:44:17.647357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:17.647329 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" event={"ID":"e68e1972-6268-4597-9520-c1a521a3d998","Type":"ContainerStarted","Data":"f296c78910f445256e50a0fe9b736a3c23134ab4d4c5e7089207174c94c996e7"} Nov 03 20:44:18.654086 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:18.654044 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" event={"ID":"e68e1972-6268-4597-9520-c1a521a3d998","Type":"ContainerStarted","Data":"8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416"} Nov 03 20:44:18.764528 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:18.764461 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podStartSLOduration=2.7644371679999997 podStartE2EDuration="2.764437168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:18.761524654 +0000 UTC m=+925.998502540" watchObservedRunningTime="2026-11-03 20:44:18.764437168 +0000 UTC m=+926.001415045" Nov 03 20:44:19.232284 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:19.232247 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:26.080984 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:44:26.080947 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:44:27.231154 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:27.231121 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:27.243203 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:27.243176 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:27.706722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:27.706696 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:44:28.758085 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.758054 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c"] Nov 03 20:44:28.762383 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.762364 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:28.859774 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.859746 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"openshift-service-ca.crt\"" Nov 03 20:44:28.860775 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.860733 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"default-dockercfg-sv9gn\"" Nov 03 20:44:28.860964 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.860785 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"kube-root-ca.crt\"" Nov 03 20:44:28.861064 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.861002 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"simulator-w3-fe2825-kserve-self-signed-certs\"" Nov 03 20:44:28.862572 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.862553 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c"] Nov 03 20:44:28.912518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.912468 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tls-certs\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:28.912860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.912839 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-model-cache\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:28.913022 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.912995 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-home\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:28.913129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.913103 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-dshm\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:28.913233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.913153 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6v6\" (UniqueName: \"kubernetes.io/projected/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-kube-api-access-9t6v6\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:28.913233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:28.913201 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tmp-dir\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.014317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.014237 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6v6\" (UniqueName: \"kubernetes.io/projected/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-kube-api-access-9t6v6\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.014317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.014290 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tmp-dir\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.014552 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.014414 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tls-certs\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.014552 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.014455 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-model-cache\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.014552 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.014480 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-home\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.014667 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.014554 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-dshm\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.014787 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.014769 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tmp-dir\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.015060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.015037 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-home\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.015060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.015054 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-model-cache\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.016919 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.016892 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-dshm\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.017131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.017110 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tls-certs\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.067130 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.067094 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6v6\" (UniqueName: \"kubernetes.io/projected/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-kube-api-access-9t6v6\") pod \"simulator-w3-fe2825-kserve-6d49744d7b-qtc4c\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.074209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.073958 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:29.454489 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.454448 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c"] Nov 03 20:44:29.461135 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:44:29.461106 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5fe195c_3ab8_4197_ae4d_5c3ca2452117.slice/crio-76e562077fb715da4e4298ecdac8b32e80a5794c38efe4146128e5745e931446 WatchSource:0}: Error finding container 76e562077fb715da4e4298ecdac8b32e80a5794c38efe4146128e5745e931446: Status 404 returned error can't find the container with id 76e562077fb715da4e4298ecdac8b32e80a5794c38efe4146128e5745e931446 Nov 03 20:44:29.706122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.706026 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" event={"ID":"e5fe195c-3ab8-4197-ae4d-5c3ca2452117","Type":"ContainerStarted","Data":"53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675"} Nov 03 20:44:29.706122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.706072 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" event={"ID":"e5fe195c-3ab8-4197-ae4d-5c3ca2452117","Type":"ContainerStarted","Data":"76e562077fb715da4e4298ecdac8b32e80a5794c38efe4146128e5745e931446"} Nov 03 20:44:29.816729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:29.816674 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podStartSLOduration=1.816661493 podStartE2EDuration="1.816661493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:29.815075602 +0000 UTC m=+937.052053486" watchObservedRunningTime="2026-11-03 20:44:29.816661493 +0000 UTC m=+937.053639366" Nov 03 20:44:30.074794 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:30.074714 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:39.074891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:39.074805 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:39.086930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:39.086906 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:39.763631 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:39.763605 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:44:44.543950 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.543909 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8"] Nov 03 20:44:44.547644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.547617 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.555505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.555468 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-model-cache\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.555653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.555510 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzt6\" (UniqueName: \"kubernetes.io/projected/aff579ec-c609-4092-b348-6f4c24e2dc93-kube-api-access-djzt6\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.555653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.555567 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-home\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.555653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.555625 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff579ec-c609-4092-b348-6f4c24e2dc93-tls-certs\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.555833 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.555694 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-dshm\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.555833 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.555733 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-tmp-dir\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.645543 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.645509 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"openshift-service-ca.crt\"" Nov 03 20:44:44.645808 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.645789 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"premium-w5-1ec892-kserve-self-signed-certs\"" Nov 03 20:44:44.645918 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.645817 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"default-dockercfg-sx87j\"" Nov 03 20:44:44.645918 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.645891 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"kube-root-ca.crt\"" Nov 03 20:44:44.647856 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.647833 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8"] Nov 03 20:44:44.656517 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656488 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-dshm\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.656629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656525 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-tmp-dir\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.656629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656565 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-model-cache\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.656629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656588 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzt6\" (UniqueName: \"kubernetes.io/projected/aff579ec-c609-4092-b348-6f4c24e2dc93-kube-api-access-djzt6\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.656783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656641 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-home\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.656783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656691 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff579ec-c609-4092-b348-6f4c24e2dc93-tls-certs\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.657013 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656990 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-tmp-dir\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.657072 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.656998 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-model-cache\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.657220 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.657197 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-home\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.658847 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.658828 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-dshm\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.659246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.659225 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff579ec-c609-4092-b348-6f4c24e2dc93-tls-certs\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.715604 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.715547 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzt6\" (UniqueName: \"kubernetes.io/projected/aff579ec-c609-4092-b348-6f4c24e2dc93-kube-api-access-djzt6\") pod \"premium-w5-1ec892-kserve-7df765b78c-bltv8\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:44.861692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:44.861610 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:45.285505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:45.285464 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8"] Nov 03 20:44:45.291850 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:44:45.291823 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaff579ec_c609_4092_b348_6f4c24e2dc93.slice/crio-0db6d2d79e7b8aca932ccf52b4a23910db73d4cec7b9d99e709855f08add42f0 WatchSource:0}: Error finding container 0db6d2d79e7b8aca932ccf52b4a23910db73d4cec7b9d99e709855f08add42f0: Status 404 returned error can't find the container with id 0db6d2d79e7b8aca932ccf52b4a23910db73d4cec7b9d99e709855f08add42f0 Nov 03 20:44:45.784650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:45.784610 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" event={"ID":"aff579ec-c609-4092-b348-6f4c24e2dc93","Type":"ContainerStarted","Data":"47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306"} Nov 03 20:44:45.785038 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:45.784664 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" event={"ID":"aff579ec-c609-4092-b348-6f4c24e2dc93","Type":"ContainerStarted","Data":"0db6d2d79e7b8aca932ccf52b4a23910db73d4cec7b9d99e709855f08add42f0"} Nov 03 20:44:45.862213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:45.862183 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:45.892372 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:45.892310 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podStartSLOduration=1.892292926 podStartE2EDuration="1.892292926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:45.890970959 +0000 UTC m=+953.127948857" watchObservedRunningTime="2026-11-03 20:44:45.892292926 +0000 UTC m=+953.129270803" Nov 03 20:44:54.861780 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:54.861746 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:54.879244 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:54.879209 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:55.841529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:44:55.841501 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:44:56.087955 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:44:56.087918 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:45:00.185728 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.185690 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v"] Nov 03 20:45:00.190396 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.190378 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.238943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.238920 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 20:45:00.301908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.301865 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdd5b4d-226c-45b9-8374-9fb6852859ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.302067 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.302013 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzdhr\" (UniqueName: \"kubernetes.io/projected/5fdd5b4d-226c-45b9-8374-9fb6852859ca-kube-api-access-qzdhr\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.376201 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.376148 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v"] Nov 03 20:45:00.403452 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.403420 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzdhr\" (UniqueName: \"kubernetes.io/projected/5fdd5b4d-226c-45b9-8374-9fb6852859ca-kube-api-access-qzdhr\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.403612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.403511 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdd5b4d-226c-45b9-8374-9fb6852859ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.404110 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.404090 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdd5b4d-226c-45b9-8374-9fb6852859ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.457271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.457213 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzdhr\" (UniqueName: \"kubernetes.io/projected/5fdd5b4d-226c-45b9-8374-9fb6852859ca-kube-api-access-qzdhr\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.503195 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.503166 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:00.881866 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:00.881782 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v"] Nov 03 20:45:00.889758 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:45:00.889724 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdd5b4d_226c_45b9_8374_9fb6852859ca.slice/crio-94e871415b273b559315b6ff12654110e3a78165d2d2e782a1b7055ae7b0229b WatchSource:0}: Error finding container 94e871415b273b559315b6ff12654110e3a78165d2d2e782a1b7055ae7b0229b: Status 404 returned error can't find the container with id 94e871415b273b559315b6ff12654110e3a78165d2d2e782a1b7055ae7b0229b Nov 03 20:45:01.861384 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:01.861336 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" event={"ID":"5fdd5b4d-226c-45b9-8374-9fb6852859ca","Type":"ContainerStarted","Data":"94e871415b273b559315b6ff12654110e3a78165d2d2e782a1b7055ae7b0229b"} Nov 03 20:45:18.944060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:18.943973 4227 generic.go:356] "Generic (PLEG): container finished" podID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerID="6b08da70de2a479ec59ff3b1d15cb42ecabf98cb8af3e3d5121cb3f85430ff7c" exitCode=60 Nov 03 20:45:18.944060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:18.944016 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" event={"ID":"5fdd5b4d-226c-45b9-8374-9fb6852859ca","Type":"ContainerDied","Data":"6b08da70de2a479ec59ff3b1d15cb42ecabf98cb8af3e3d5121cb3f85430ff7c"} Nov 03 20:45:18.944529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:18.944236 4227 scope.go:122] "RemoveContainer" containerID="6b08da70de2a479ec59ff3b1d15cb42ecabf98cb8af3e3d5121cb3f85430ff7c" Nov 03 20:45:19.951053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:19.951019 4227 generic.go:356] "Generic (PLEG): container finished" podID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerID="c7f76885da6b771601ab28f6b3894a7456578c4ef7eaf2c97a56ea1d576e7d48" exitCode=60 Nov 03 20:45:19.951459 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:19.951105 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" event={"ID":"5fdd5b4d-226c-45b9-8374-9fb6852859ca","Type":"ContainerDied","Data":"c7f76885da6b771601ab28f6b3894a7456578c4ef7eaf2c97a56ea1d576e7d48"} Nov 03 20:45:19.951459 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:19.951207 4227 scope.go:122] "RemoveContainer" containerID="6b08da70de2a479ec59ff3b1d15cb42ecabf98cb8af3e3d5121cb3f85430ff7c" Nov 03 20:45:19.951459 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:19.951451 4227 scope.go:122] "RemoveContainer" containerID="c7f76885da6b771601ab28f6b3894a7456578c4ef7eaf2c97a56ea1d576e7d48" Nov 03 20:45:19.951702 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:45:19.951679 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v_odh-ai-gateway-infra(5fdd5b4d-226c-45b9-8374-9fb6852859ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" Nov 03 20:45:20.956922 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:20.956893 4227 scope.go:122] "RemoveContainer" containerID="c7f76885da6b771601ab28f6b3894a7456578c4ef7eaf2c97a56ea1d576e7d48" Nov 03 20:45:20.957310 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:45:20.957116 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v_odh-ai-gateway-infra(5fdd5b4d-226c-45b9-8374-9fb6852859ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" Nov 03 20:45:26.080906 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:45:26.080862 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:45:33.893032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:33.892999 4227 scope.go:122] "RemoveContainer" containerID="c7f76885da6b771601ab28f6b3894a7456578c4ef7eaf2c97a56ea1d576e7d48" Nov 03 20:45:35.031655 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:35.031624 4227 generic.go:356] "Generic (PLEG): container finished" podID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerID="47880abfd8aee4a85c57c9e734754b26112acbf13534e150777554c0ac1634ae" exitCode=60 Nov 03 20:45:35.032086 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:35.031698 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" event={"ID":"5fdd5b4d-226c-45b9-8374-9fb6852859ca","Type":"ContainerDied","Data":"47880abfd8aee4a85c57c9e734754b26112acbf13534e150777554c0ac1634ae"} Nov 03 20:45:35.032086 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:35.031754 4227 scope.go:122] "RemoveContainer" containerID="c7f76885da6b771601ab28f6b3894a7456578c4ef7eaf2c97a56ea1d576e7d48" Nov 03 20:45:35.032086 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:35.032054 4227 scope.go:122] "RemoveContainer" containerID="47880abfd8aee4a85c57c9e734754b26112acbf13534e150777554c0ac1634ae" Nov 03 20:45:35.032311 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:45:35.032290 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v_odh-ai-gateway-infra(5fdd5b4d-226c-45b9-8374-9fb6852859ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" Nov 03 20:45:36.148813 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.148784 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v"] Nov 03 20:45:36.281498 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.281466 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:36.382970 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.382931 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdd5b4d-226c-45b9-8374-9fb6852859ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdd5b4d-226c-45b9-8374-9fb6852859ca-maas-api-service-ca\") pod \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " Nov 03 20:45:36.383123 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.383013 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdd5b4d-226c-45b9-8374-9fb6852859ca-kube-api-access-qzdhr\" (UniqueName: \"kubernetes.io/projected/5fdd5b4d-226c-45b9-8374-9fb6852859ca-kube-api-access-qzdhr\") pod \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\" (UID: \"5fdd5b4d-226c-45b9-8374-9fb6852859ca\") " Nov 03 20:45:36.383310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.383282 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdd5b4d-226c-45b9-8374-9fb6852859ca-maas-api-service-ca" pod "5fdd5b4d-226c-45b9-8374-9fb6852859ca" (UID: "5fdd5b4d-226c-45b9-8374-9fb6852859ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:36.385020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.384993 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdd5b4d-226c-45b9-8374-9fb6852859ca-kube-api-access-qzdhr" pod "5fdd5b4d-226c-45b9-8374-9fb6852859ca" (UID: "5fdd5b4d-226c-45b9-8374-9fb6852859ca"). InnerVolumeSpecName "kube-api-access-qzdhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:36.484739 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.484706 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdd5b4d-226c-45b9-8374-9fb6852859ca-maas-api-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:36.484739 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:36.484734 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzdhr\" (UniqueName: \"kubernetes.io/projected/5fdd5b4d-226c-45b9-8374-9fb6852859ca-kube-api-access-qzdhr\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:37.044068 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:37.044040 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" Nov 03 20:45:37.044252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:37.044078 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v" event={"ID":"5fdd5b4d-226c-45b9-8374-9fb6852859ca","Type":"ContainerDied","Data":"94e871415b273b559315b6ff12654110e3a78165d2d2e782a1b7055ae7b0229b"} Nov 03 20:45:37.044252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:37.044135 4227 scope.go:122] "RemoveContainer" containerID="47880abfd8aee4a85c57c9e734754b26112acbf13534e150777554c0ac1634ae" Nov 03 20:45:37.198692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:37.198614 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v"] Nov 03 20:45:37.244650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:37.244617 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895645-q785v"] Nov 03 20:45:38.898852 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:38.898782 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" path="/var/lib/kubelet/pods/5fdd5b4d-226c-45b9-8374-9fb6852859ca/volumes" Nov 03 20:45:45.082905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.082869 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492"] Nov 03 20:45:45.083445 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083426 4227 cpu_manager.go:414] "containerMap: removing container" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.083499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083448 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.083499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083467 4227 cpu_manager.go:414] "containerMap: removing container" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.083499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083475 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.083499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083497 4227 cpu_manager.go:414] "containerMap: removing container" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.083622 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083505 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.083622 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083612 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.083679 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.083630 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:45:45.090890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.090868 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.139684 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.139655 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492"] Nov 03 20:45:45.169708 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.169675 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-model-cache\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.169865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.169742 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-home\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.169865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.169846 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f781fc90-550e-4649-b618-a1375b3cfbb8-tls-certs\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.169976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.169953 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-tmp-dir\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.170023 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.170004 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2vl\" (UniqueName: \"kubernetes.io/projected/f781fc90-550e-4649-b618-a1375b3cfbb8-kube-api-access-sx2vl\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.170181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.170149 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-dshm\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.184048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.184020 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"premium-w3-fe2825-kserve-self-signed-certs\"" Nov 03 20:45:45.270829 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.270790 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-dshm\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.270829 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.270844 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-model-cache\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.271092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.270902 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-home\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.271092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.270939 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f781fc90-550e-4649-b618-a1375b3cfbb8-tls-certs\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.271092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.271004 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-tmp-dir\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.271092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.271058 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2vl\" (UniqueName: \"kubernetes.io/projected/f781fc90-550e-4649-b618-a1375b3cfbb8-kube-api-access-sx2vl\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.271682 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.271644 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-model-cache\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.271903 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.271883 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-home\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.272822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.272800 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-tmp-dir\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.276629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.276605 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-dshm\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.276997 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.276971 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f781fc90-550e-4649-b618-a1375b3cfbb8-tls-certs\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.326275 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.326243 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2vl\" (UniqueName: \"kubernetes.io/projected/f781fc90-550e-4649-b618-a1375b3cfbb8-kube-api-access-sx2vl\") pod \"premium-w3-fe2825-kserve-66b8989765-tm492\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.402922 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.402880 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:45.814260 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:45.814186 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492"] Nov 03 20:45:45.821256 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:45:45.821231 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf781fc90_550e_4649_b618_a1375b3cfbb8.slice/crio-904b94dfe5326c194ec6184766b364ee87738d978ffb15782d0b77d03270447a WatchSource:0}: Error finding container 904b94dfe5326c194ec6184766b364ee87738d978ffb15782d0b77d03270447a: Status 404 returned error can't find the container with id 904b94dfe5326c194ec6184766b364ee87738d978ffb15782d0b77d03270447a Nov 03 20:45:46.091230 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:46.091113 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" event={"ID":"f781fc90-550e-4649-b618-a1375b3cfbb8","Type":"ContainerStarted","Data":"44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67"} Nov 03 20:45:46.091230 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:46.091178 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" event={"ID":"f781fc90-550e-4649-b618-a1375b3cfbb8","Type":"ContainerStarted","Data":"904b94dfe5326c194ec6184766b364ee87738d978ffb15782d0b77d03270447a"} Nov 03 20:45:46.201841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:46.201687 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podStartSLOduration=1.201670237 podStartE2EDuration="1.201670237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:46.201471502 +0000 UTC m=+1013.438449377" watchObservedRunningTime="2026-11-03 20:45:46.201670237 +0000 UTC m=+1013.438648121" Nov 03 20:45:46.403038 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:46.403013 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:55.403981 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:55.403947 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:55.416369 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:55.416346 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:45:56.081061 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:45:56.081025 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:45:56.148270 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:45:56.148240 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:46:20.525994 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:20.525955 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5"] Nov 03 20:46:20.526490 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:20.526145 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" containerID="cri-o://4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da" gracePeriod=30 Nov 03 20:46:20.572808 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:20.572777 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6"] Nov 03 20:46:20.572977 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:20.572957 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" containerID="cri-o://c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3" gracePeriod=30 Nov 03 20:46:25.826180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.826130 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:46:25.851845 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.851820 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:46:25.902176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902136 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-socket\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902217 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-data\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902245 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-credential-socket\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902282 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-kube-api-access-ghpht\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-kube-api-access-ghpht\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902307 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-envoy\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902333 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-certs\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902361 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-istio-token\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-istio-token\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902390 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239b1294-920b-4389-a804-2a31f2418af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b1294-920b-4389-a804-2a31f2418af9-istiod-ca-cert\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902428 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-data\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902424 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-socket" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.902488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902459 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-certs\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902490 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8718f11f-7b9d-4097-8d93-98c9195ab382-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718f11f-7b9d-4097-8d93-98c9195ab382-istiod-ca-cert\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902532 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-data" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902551 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-token\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-token\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902551 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-credential-socket" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902614 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-envoy\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902648 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/239b1294-920b-4389-a804-2a31f2418af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b1294-920b-4389-a804-2a31f2418af9-istio-podinfo\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902723 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-socket\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902736 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-certs" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902772 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-podinfo\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.902835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902801 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-credential-socket\") pod \"8718f11f-7b9d-4097-8d93-98c9195ab382\" (UID: \"8718f11f-7b9d-4097-8d93-98c9195ab382\") " Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902846 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-kube-api-access-4dngm\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-kube-api-access-4dngm\") pod \"239b1294-920b-4389-a804-2a31f2418af9\" (UID: \"239b1294-920b-4389-a804-2a31f2418af9\") " Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.902916 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-data" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903097 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-certs" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903259 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903279 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903293 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903308 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903320 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.903350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903333 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:25.903822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903422 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239b1294-920b-4389-a804-2a31f2418af9-istiod-ca-cert" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:25.903822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903545 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-socket" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.904358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.903949 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8718f11f-7b9d-4097-8d93-98c9195ab382-istiod-ca-cert" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:25.904358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.904184 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-credential-socket" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.905000 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.904973 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-istio-token" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:25.905403 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.905376 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-token" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:25.905491 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.905468 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-kube-api-access-4dngm" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "kube-api-access-4dngm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:25.905912 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.905895 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/239b1294-920b-4389-a804-2a31f2418af9-istio-podinfo" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:46:25.906259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.906227 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-envoy" pod "239b1294-920b-4389-a804-2a31f2418af9" (UID: "239b1294-920b-4389-a804-2a31f2418af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.906825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.906808 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-envoy" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:25.906931 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.906913 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-podinfo" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:46:25.907106 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:25.907091 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-kube-api-access-ghpht" pod "8718f11f-7b9d-4097-8d93-98c9195ab382" (UID: "8718f11f-7b9d-4097-8d93-98c9195ab382"). InnerVolumeSpecName "kube-api-access-ghpht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:26.004842 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004808 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.004842 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004841 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004855 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b1294-920b-4389-a804-2a31f2418af9-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004870 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004884 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718f11f-7b9d-4097-8d93-98c9195ab382-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004897 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718f11f-7b9d-4097-8d93-98c9195ab382-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004911 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dngm\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-kube-api-access-4dngm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004926 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghpht\" (UniqueName: \"kubernetes.io/projected/8718f11f-7b9d-4097-8d93-98c9195ab382-kube-api-access-ghpht\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004941 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b1294-920b-4389-a804-2a31f2418af9-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004953 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b1294-920b-4389-a804-2a31f2418af9-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004967 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b1294-920b-4389-a804-2a31f2418af9-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.005092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.004980 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718f11f-7b9d-4097-8d93-98c9195ab382-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.083929 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:46:26.083854 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:46:26.313243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.313203 4227 generic.go:356] "Generic (PLEG): container finished" podID="239b1294-920b-4389-a804-2a31f2418af9" containerID="4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da" exitCode=0 Nov 03 20:46:26.313476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.313255 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" Nov 03 20:46:26.313476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.313289 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" event={"ID":"239b1294-920b-4389-a804-2a31f2418af9","Type":"ContainerDied","Data":"4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da"} Nov 03 20:46:26.313476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.313338 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5" event={"ID":"239b1294-920b-4389-a804-2a31f2418af9","Type":"ContainerDied","Data":"a213fd8061dce73f88e1f09fa6ada54516a4fafccdc9beb8a826936d57bd0168"} Nov 03 20:46:26.313476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.313367 4227 scope.go:122] "RemoveContainer" containerID="4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da" Nov 03 20:46:26.314956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.314918 4227 generic.go:356] "Generic (PLEG): container finished" podID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerID="c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3" exitCode=0 Nov 03 20:46:26.314956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.314951 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" event={"ID":"8718f11f-7b9d-4097-8d93-98c9195ab382","Type":"ContainerDied","Data":"c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3"} Nov 03 20:46:26.315131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.315005 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" event={"ID":"8718f11f-7b9d-4097-8d93-98c9195ab382","Type":"ContainerDied","Data":"b9aea9d7ef9efca60aaef973f0726ba387c0a5fcf53a528f3175f19fb05794d3"} Nov 03 20:46:26.315131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.315027 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6" Nov 03 20:46:26.323756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.323734 4227 scope.go:122] "RemoveContainer" containerID="4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da" Nov 03 20:46:26.324016 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:46:26.323999 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da\": container with ID starting with 4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da not found: ID does not exist" containerID="4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da" Nov 03 20:46:26.324066 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.324024 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da"} err="failed to get container status \"4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da\": rpc error: code = NotFound desc = could not find container \"4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da\": container with ID starting with 4c98f3742704c633fd9e574740cdd64e67e6957aadf56027858ef9da306bc9da not found: ID does not exist" Nov 03 20:46:26.324066 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.324042 4227 scope.go:122] "RemoveContainer" containerID="c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3" Nov 03 20:46:26.331983 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.331967 4227 scope.go:122] "RemoveContainer" containerID="c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3" Nov 03 20:46:26.332285 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:46:26.332266 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3\": container with ID starting with c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3 not found: ID does not exist" containerID="c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3" Nov 03 20:46:26.332358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.332289 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3"} err="failed to get container status \"c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3\": rpc error: code = NotFound desc = could not find container \"c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3\": container with ID starting with c69429cf1da741931a2224ba76363c7137500bbe1e32dcb14b6edb9f21ff99c3 not found: ID does not exist" Nov 03 20:46:26.467352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.467326 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6"] Nov 03 20:46:26.514088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.514062 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-h62q6"] Nov 03 20:46:26.572957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.572916 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5"] Nov 03 20:46:26.579474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.579448 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p64p5"] Nov 03 20:46:26.898925 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.898833 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239b1294-920b-4389-a804-2a31f2418af9" path="/var/lib/kubelet/pods/239b1294-920b-4389-a804-2a31f2418af9/volumes" Nov 03 20:46:26.899394 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:26.899291 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" path="/var/lib/kubelet/pods/8718f11f-7b9d-4097-8d93-98c9195ab382/volumes" Nov 03 20:46:39.768586 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.768549 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2"] Nov 03 20:46:39.769759 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.769730 4227 cpu_manager.go:414] "containerMap: removing container" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" Nov 03 20:46:39.769890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.769761 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" Nov 03 20:46:39.769890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.769801 4227 cpu_manager.go:414] "containerMap: removing container" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" Nov 03 20:46:39.769890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.769809 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" Nov 03 20:46:39.770057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.769952 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdd5b4d-226c-45b9-8374-9fb6852859ca" containerName="cleanup" Nov 03 20:46:39.770057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.769969 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="239b1294-920b-4389-a804-2a31f2418af9" containerName="istio-proxy" Nov 03 20:46:39.770057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.769985 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="8718f11f-7b9d-4097-8d93-98c9195ab382" containerName="istio-proxy" Nov 03 20:46:39.773375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.773353 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.822061 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.822036 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a19980-openshift-default-dockercfg-nbrs7\"" Nov 03 20:46:39.837879 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.837852 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.837900 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.837987 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838090 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.838039 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838090 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.838067 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838090 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.838087 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.838135 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.838150 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqh9p\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-kube-api-access-vqh9p\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.838214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.838199 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.870995 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.870969 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2"] Nov 03 20:46:39.938732 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.938705 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.938839 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.938749 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.938839 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.938772 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.938839 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.938791 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.938962 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.938841 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.938962 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.938856 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqh9p\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-kube-api-access-vqh9p\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939058 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939023 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939109 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939149 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939432 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939406 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939517 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939435 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939517 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939454 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939517 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939512 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.939682 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.939646 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.941202 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.941179 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.941300 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.941261 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.991723 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.991695 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqh9p\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-kube-api-access-vqh9p\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:39.992031 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:39.992007 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:40.089743 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:40.089562 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:40.515355 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:40.515321 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2"] Nov 03 20:46:40.521861 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:46:40.521835 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968f7168_1fe9_4ff6_a5a3_3d4e16d97207.slice/crio-708007e531682b0936a8a5284af77ea77eaef6f06e86c2ed49e29d5d16597d62 WatchSource:0}: Error finding container 708007e531682b0936a8a5284af77ea77eaef6f06e86c2ed49e29d5d16597d62: Status 404 returned error can't find the container with id 708007e531682b0936a8a5284af77ea77eaef6f06e86c2ed49e29d5d16597d62 Nov 03 20:46:40.524117 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:40.524087 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:40.524216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:40.524150 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:40.524216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:40.524195 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:41.390744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:41.390704 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" event={"ID":"968f7168-1fe9-4ff6-a5a3-3d4e16d97207","Type":"ContainerStarted","Data":"837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c"} Nov 03 20:46:41.391123 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:41.390761 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" event={"ID":"968f7168-1fe9-4ff6-a5a3-3d4e16d97207","Type":"ContainerStarted","Data":"708007e531682b0936a8a5284af77ea77eaef6f06e86c2ed49e29d5d16597d62"} Nov 03 20:46:41.612977 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:41.612927 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podStartSLOduration=2.612913631 podStartE2EDuration="2.612913631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:41.565783905 +0000 UTC m=+1068.802761776" watchObservedRunningTime="2026-11-03 20:46:41.612913631 +0000 UTC m=+1068.849891504" Nov 03 20:46:42.090216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:42.090181 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:42.091567 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:42.091546 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:46:42.091616 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:42.091594 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:46:43.090131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:43.090097 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:46:43.090543 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:43.090151 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:46:44.090295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:44.090260 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:46:44.090768 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:44.090353 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:46:45.090666 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:45.090624 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:46:45.091175 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:45.090675 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:46:46.090504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:46.090469 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:46:46.090679 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:46.090526 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:46:47.097017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:47.096981 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:47.098014 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:47.097988 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:47.098696 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:47.098669 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 20:46:54.531931 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.531897 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx"] Nov 03 20:46:54.545934 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.545904 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.590561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.590528 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx"] Nov 03 20:46:54.608712 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.608690 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.608835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.608719 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.608835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.608741 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.608835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.608820 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.608957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.608938 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.609020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.609005 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.609064 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.609048 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.609098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.609092 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.609149 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.609137 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5v4\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-kube-api-access-2h5v4\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.634237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.634216 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-984cbc-openshift-default-dockercfg-qwh2c\"" Nov 03 20:46:54.709868 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.709847 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.709999 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.709876 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.709999 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.709897 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710087 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710015 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5v4\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-kube-api-access-2h5v4\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710091 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710113 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710134 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710180 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710215 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710392 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710245 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710392 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710301 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710510 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710486 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710543 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.710882 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.710862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.711898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.711878 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.712605 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.712588 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.762824 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.762803 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.763026 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.762995 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5v4\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-kube-api-access-2h5v4\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:54.861312 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:54.861264 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:55.282861 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.282822 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx"] Nov 03 20:46:55.289065 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:46:55.289028 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6751b5ae_fbb7_4529_bbb5_be422bdeb53e.slice/crio-48a05f0e71fb2df17ceb5ff38d196381e4c07875ede5ab640eba3f456ac67087 WatchSource:0}: Error finding container 48a05f0e71fb2df17ceb5ff38d196381e4c07875ede5ab640eba3f456ac67087: Status 404 returned error can't find the container with id 48a05f0e71fb2df17ceb5ff38d196381e4c07875ede5ab640eba3f456ac67087 Nov 03 20:46:55.291422 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.291390 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:55.291504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.291455 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:55.291504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.291484 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:55.461476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.461431 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" event={"ID":"6751b5ae-fbb7-4529-bbb5-be422bdeb53e","Type":"ContainerStarted","Data":"734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660"} Nov 03 20:46:55.461655 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.461502 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" event={"ID":"6751b5ae-fbb7-4529-bbb5-be422bdeb53e","Type":"ContainerStarted","Data":"48a05f0e71fb2df17ceb5ff38d196381e4c07875ede5ab640eba3f456ac67087"} Nov 03 20:46:55.572204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.572096 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podStartSLOduration=1.572082118 podStartE2EDuration="1.572082118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:55.570976847 +0000 UTC m=+1082.807954720" watchObservedRunningTime="2026-11-03 20:46:55.572082118 +0000 UTC m=+1082.809059991" Nov 03 20:46:55.863238 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:55.862244 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:46:56.100033 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:46:56.099999 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:46:56.866033 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:56.865997 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:46:56.871632 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:56.871246 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:46:57.863484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:57.863449 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:46:57.863791 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:57.863765 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:46:58.862891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:58.862465 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:46:58.862891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:58.862534 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:46:59.868184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:59.863459 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:46:59.868184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:46:59.863528 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:47:00.872277 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:00.872242 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:47:00.872885 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:00.872852 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:47:01.866415 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:01.866382 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:47:01.866702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:01.866680 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:47:01.867298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:01.867281 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 20:47:05.864979 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.864942 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f"] Nov 03 20:47:05.868768 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.868744 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.911420 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.911389 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b"] Nov 03 20:47:05.914803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.914787 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:05.938341 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938318 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938349 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938386 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd4de8-8914-4263-8300-70c81a055876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938587 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938476 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938587 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938561 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938685 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938597 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938685 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938645 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvl55\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-kube-api-access-zvl55\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938771 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938752 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.938829 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.938819 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd4de8-8914-4263-8300-70c81a055876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:05.964047 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:05.964021 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f"] Nov 03 20:47:06.012674 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.012642 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b"] Nov 03 20:47:06.040091 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040056 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.040091 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040091 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be870e-37e3-4a37-80cd-f4a2642ee303-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.040303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040112 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.040303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040248 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd4de8-8914-4263-8300-70c81a055876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.040303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040272 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.040303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040295 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.040430 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040389 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.040430 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040411 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.040525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040449 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd4de8-8914-4263-8300-70c81a055876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.040580 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040533 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.040580 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040561 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.040756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040719 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95wj\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-kube-api-access-m95wj\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.040898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040777 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.040898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040867 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.041021 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040913 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.041021 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040935 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.041021 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.040973 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvl55\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-kube-api-access-zvl55\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.041021 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.041006 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.041261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.041075 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.041261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.041078 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.041261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.041128 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.041261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.041185 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd4de8-8914-4263-8300-70c81a055876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.041397 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.041310 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.042756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.042733 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.042845 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.042827 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd4de8-8914-4263-8300-70c81a055876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.094279 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.094241 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.094422 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.094355 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvl55\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-kube-api-access-zvl55\") pod \"maas-default-gateway-openshift-default-696f47c6c-jck8f\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142366 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142423 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142486 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142520 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95wj\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-kube-api-access-m95wj\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142582 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142658 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142721 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142745 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be870e-37e3-4a37-80cd-f4a2642ee303-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.142776 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143876 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.143309 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.143876 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.143608 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.144201 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.143941 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.146362 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.145938 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be870e-37e3-4a37-80cd-f4a2642ee303-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.146792 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.146747 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.147770 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.147741 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.148644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.148621 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.184079 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.184051 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:06.197691 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.197667 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95wj\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-kube-api-access-m95wj\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.198079 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.198058 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qtq8b\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.227259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.227228 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:06.608880 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.608844 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f"] Nov 03 20:47:06.611298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.611264 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b"] Nov 03 20:47:06.615622 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:47:06.615567 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcd4de8_8914_4263_8300_70c81a055876.slice/crio-288f582994b743253667cd62c701026c3fe77bbf4a9255ac8a691d64e403fa62 WatchSource:0}: Error finding container 288f582994b743253667cd62c701026c3fe77bbf4a9255ac8a691d64e403fa62: Status 404 returned error can't find the container with id 288f582994b743253667cd62c701026c3fe77bbf4a9255ac8a691d64e403fa62 Nov 03 20:47:06.617556 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:47:06.617529 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3be870e_37e3_4a37_80cd_f4a2642ee303.slice/crio-7d0b21f786101bb1f45f4f879a8dc686c9f39644e0ae7febe4b59f94e9e13a1d WatchSource:0}: Error finding container 7d0b21f786101bb1f45f4f879a8dc686c9f39644e0ae7febe4b59f94e9e13a1d: Status 404 returned error can't find the container with id 7d0b21f786101bb1f45f4f879a8dc686c9f39644e0ae7febe4b59f94e9e13a1d Nov 03 20:47:06.618019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.617985 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:06.618100 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.618063 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:06.618178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.618133 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:06.619653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.619626 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:06.619751 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.619687 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:06.619751 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:06.619731 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:07.710198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:07.710140 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" event={"ID":"b3be870e-37e3-4a37-80cd-f4a2642ee303","Type":"ContainerStarted","Data":"00ce6e13a42bdec6b4fddf7fcab2c277381377591f4e755c898447eb83461b34"} Nov 03 20:47:07.710797 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:07.710767 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" event={"ID":"b3be870e-37e3-4a37-80cd-f4a2642ee303","Type":"ContainerStarted","Data":"7d0b21f786101bb1f45f4f879a8dc686c9f39644e0ae7febe4b59f94e9e13a1d"} Nov 03 20:47:07.728911 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:07.728881 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" event={"ID":"9bcd4de8-8914-4263-8300-70c81a055876","Type":"ContainerStarted","Data":"ddf9f18f2d6d96d9aa5c0817353f87d80f605b9bbe49984f3d6112bdd2082e0e"} Nov 03 20:47:07.729136 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:07.729119 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" event={"ID":"9bcd4de8-8914-4263-8300-70c81a055876","Type":"ContainerStarted","Data":"288f582994b743253667cd62c701026c3fe77bbf4a9255ac8a691d64e403fa62"} Nov 03 20:47:07.939923 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:07.939854 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podStartSLOduration=2.939831913 podStartE2EDuration="2.939831913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:07.88663634 +0000 UTC m=+1095.123614217" watchObservedRunningTime="2026-11-03 20:47:07.939831913 +0000 UTC m=+1095.176809791" Nov 03 20:47:08.188207 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:08.188153 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:08.188570 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:08.188542 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jck8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:47:08.188749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:08.188722 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:47:08.231668 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:08.231628 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:08.232016 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:08.231990 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qtq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:47:08.232216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:08.232189 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:47:09.187183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:09.186818 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jck8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:47:09.187183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:09.186899 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:47:09.227664 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:09.227608 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qtq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:47:09.227925 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:09.227675 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:47:10.189541 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:10.189500 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jck8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:47:10.190143 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:10.190113 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:47:10.228633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:10.228597 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qtq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:47:10.228844 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:10.228824 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:47:11.189183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:11.184602 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jck8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:47:11.189183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:11.184677 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:47:11.227779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:11.227747 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qtq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:47:11.228265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:11.228239 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:47:12.185185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:12.184565 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jck8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:47:12.185185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:12.184627 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:47:12.228077 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:12.228044 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qtq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:47:12.228472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:12.228088 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:47:13.188198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:13.188149 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jck8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:47:13.188503 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:13.188472 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:47:13.227656 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:13.227625 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qtq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:47:13.227838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:13.227685 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:47:14.185387 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:14.185342 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jck8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:47:14.185876 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:14.185413 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:47:14.227959 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:14.227917 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qtq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:47:14.228122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:14.227988 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:47:15.650023 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:15.649987 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:15.650719 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:15.650698 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:15.651751 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:15.651724 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:47:15.711611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:15.711573 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:15.712681 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:15.712650 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:15.713515 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:15.713498 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:47:15.846100 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:15.846037 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podStartSLOduration=10.846017943 podStartE2EDuration="10.846017943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:08.020940379 +0000 UTC m=+1095.257918278" watchObservedRunningTime="2026-11-03 20:47:15.846017943 +0000 UTC m=+1103.082995818" Nov 03 20:47:21.460396 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.460360 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5"] Nov 03 20:47:21.463967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.463946 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.524740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.524708 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-model-cache\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.524924 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.524792 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzd2\" (UniqueName: \"kubernetes.io/projected/e0ad67a2-6719-4c18-9f54-fc3633bccc78-kube-api-access-kjzd2\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.524924 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.524888 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tmp-dir\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.525044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.524939 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-dshm\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.525044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.525008 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-home\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.525182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.525041 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tls-certs\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.562093 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.562066 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"kube-root-ca.crt\"" Nov 03 20:47:21.562364 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.562335 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"default-dockercfg-zj2dk\"" Nov 03 20:47:21.562540 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.562394 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"openshift-service-ca.crt\"" Nov 03 20:47:21.562611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.562597 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a19980\"/\"distinct-w6-a19980-kserve-self-signed-certs\"" Nov 03 20:47:21.564970 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.564946 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5"] Nov 03 20:47:21.626479 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626452 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-dshm\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.626644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626492 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-home\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.626644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626524 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tls-certs\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.626644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626579 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-model-cache\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.626849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626655 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzd2\" (UniqueName: \"kubernetes.io/projected/e0ad67a2-6719-4c18-9f54-fc3633bccc78-kube-api-access-kjzd2\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.626849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626706 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tmp-dir\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.626959 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626924 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-home\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.627023 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.626999 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-model-cache\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.627081 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.627043 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tmp-dir\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.628778 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.628760 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-dshm\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.629151 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.629129 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tls-certs\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.694288 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.694219 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzd2\" (UniqueName: \"kubernetes.io/projected/e0ad67a2-6719-4c18-9f54-fc3633bccc78-kube-api-access-kjzd2\") pod \"distinct-w6-a19980-kserve-6649db57f9-v2tm5\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:21.780721 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:21.780639 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:22.212594 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:22.212561 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5"] Nov 03 20:47:22.218923 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:47:22.218900 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0ad67a2_6719_4c18_9f54_fc3633bccc78.slice/crio-aa1010953088d4216ae7670e40d7b2d3b5c2a6ebd0f069019044ac9b706bf2ca WatchSource:0}: Error finding container aa1010953088d4216ae7670e40d7b2d3b5c2a6ebd0f069019044ac9b706bf2ca: Status 404 returned error can't find the container with id aa1010953088d4216ae7670e40d7b2d3b5c2a6ebd0f069019044ac9b706bf2ca Nov 03 20:47:22.825779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:22.825736 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" event={"ID":"e0ad67a2-6719-4c18-9f54-fc3633bccc78","Type":"ContainerStarted","Data":"4760c0e10469cba29c4d11b7751d2ec64762e5e983be0292cbf6b66b08076767"} Nov 03 20:47:22.826184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:22.825789 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" event={"ID":"e0ad67a2-6719-4c18-9f54-fc3633bccc78","Type":"ContainerStarted","Data":"aa1010953088d4216ae7670e40d7b2d3b5c2a6ebd0f069019044ac9b706bf2ca"} Nov 03 20:47:22.938221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:22.938155 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podStartSLOduration=1.938141802 podStartE2EDuration="1.938141802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:22.935980673 +0000 UTC m=+1110.172958549" watchObservedRunningTime="2026-11-03 20:47:22.938141802 +0000 UTC m=+1110.175119704" Nov 03 20:47:23.781711 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:23.781677 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:24.011918 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.011882 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb"] Nov 03 20:47:24.015982 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.015959 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.064817 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.064732 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1ec892-openshift-default-dockercfg-dds29\"" Nov 03 20:47:24.071216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071181 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptbd\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-kube-api-access-pptbd\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071228 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071319 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35251231-f967-4184-970b-bcb4bf41a0f5-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071421 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071385 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071426 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071535 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071554 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071619 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35251231-f967-4184-970b-bcb4bf41a0f5-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.071653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.071646 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.113580 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.113549 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb"] Nov 03 20:47:24.172693 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.172659 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35251231-f967-4184-970b-bcb4bf41a0f5-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.172857 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.172707 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.172857 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.172758 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptbd\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-kube-api-access-pptbd\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.172857 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.172783 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.172857 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.172836 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35251231-f967-4184-970b-bcb4bf41a0f5-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173099 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.172873 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173099 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.172927 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173099 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.173007 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173099 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.173034 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173325 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.173235 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173383 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.173357 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173470 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.173452 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173535 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.173518 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35251231-f967-4184-970b-bcb4bf41a0f5-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.173584 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.173566 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.175293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.175272 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.175393 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.175295 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35251231-f967-4184-970b-bcb4bf41a0f5-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.226371 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.226336 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.226371 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.226362 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptbd\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-kube-api-access-pptbd\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.329357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.329277 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:24.553459 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.553425 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z"] Nov 03 20:47:24.557438 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.557415 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.612088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.611999 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z"] Nov 03 20:47:24.678741 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.678713 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.678908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.678755 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.678908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.678773 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b472214b-3588-487d-be6e-2004270492cb-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.678908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.678861 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7nk\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-kube-api-access-qj7nk\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.679088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.678971 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.679088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.679024 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.679202 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.679104 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.679202 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.679134 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b472214b-3588-487d-be6e-2004270492cb-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.679325 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.679264 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.778582 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.778550 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb"] Nov 03 20:47:24.780834 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.780806 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.780976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.780858 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.780976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.780882 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b472214b-3588-487d-be6e-2004270492cb-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.780976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.780900 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7nk\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-kube-api-access-qj7nk\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.780976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.780935 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.780976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.780971 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781010 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781044 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b472214b-3588-487d-be6e-2004270492cb-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781092 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781405 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781274 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781469 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781443 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781657 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781597 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781785 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781713 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.781926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.781907 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b472214b-3588-487d-be6e-2004270492cb-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.783402 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.783372 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.784054 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.784029 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b472214b-3588-487d-be6e-2004270492cb-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.785825 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:47:24.785802 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35251231_f967_4184_970b_bcb4bf41a0f5.slice/crio-c763ba7304c1ddddcdf5724043fb24828d7c9c62ef8af3f01f97a26a7fa439b1 WatchSource:0}: Error finding container c763ba7304c1ddddcdf5724043fb24828d7c9c62ef8af3f01f97a26a7fa439b1: Status 404 returned error can't find the container with id c763ba7304c1ddddcdf5724043fb24828d7c9c62ef8af3f01f97a26a7fa439b1 Nov 03 20:47:24.788193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.788145 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:24.788294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.788240 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:24.788354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.788304 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:24.834228 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.834204 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.838131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.838102 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7nk\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-kube-api-access-qj7nk\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:24.853953 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.853918 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" event={"ID":"35251231-f967-4184-970b-bcb4bf41a0f5","Type":"ContainerStarted","Data":"c763ba7304c1ddddcdf5724043fb24828d7c9c62ef8af3f01f97a26a7fa439b1"} Nov 03 20:47:24.871212 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:24.871147 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:25.317211 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.317177 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z"] Nov 03 20:47:25.325462 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:47:25.325433 4227 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb472214b_3588_487d_be6e_2004270492cb.slice/crio-bea9acd60c785c378cc6777309808bae07230979fe69cc4fea2a6a986f3bcac8\": RecentStats: unable to find data in memory cache]" Nov 03 20:47:25.329358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.329326 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:25.329476 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.329431 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:25.329536 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.329477 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:25.877185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.877026 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" event={"ID":"b472214b-3588-487d-be6e-2004270492cb","Type":"ContainerStarted","Data":"458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc"} Nov 03 20:47:25.877185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.877101 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" event={"ID":"b472214b-3588-487d-be6e-2004270492cb","Type":"ContainerStarted","Data":"bea9acd60c785c378cc6777309808bae07230979fe69cc4fea2a6a986f3bcac8"} Nov 03 20:47:25.881180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.879892 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" event={"ID":"35251231-f967-4184-970b-bcb4bf41a0f5","Type":"ContainerStarted","Data":"c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d"} Nov 03 20:47:25.987849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:25.987794 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podStartSLOduration=1.9877793700000002 podStartE2EDuration="1.98777937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:25.985775898 +0000 UTC m=+1113.222753773" watchObservedRunningTime="2026-11-03 20:47:25.98777937 +0000 UTC m=+1113.224757243" Nov 03 20:47:26.098205 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:26.097835 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podStartSLOduration=3.09781669 podStartE2EDuration="3.09781669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:26.096337931 +0000 UTC m=+1113.333315805" watchObservedRunningTime="2026-11-03 20:47:26.09781669 +0000 UTC m=+1113.334794568" Nov 03 20:47:26.111417 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:47:26.111388 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:47:26.330498 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:26.330466 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:26.331963 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:26.331932 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 20:47:26.332096 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:26.331986 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 20:47:26.876336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:26.875873 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:26.876336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:26.876006 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 20:47:26.876336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:26.876051 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 20:47:27.329847 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:27.329809 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 20:47:27.330058 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:27.329874 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 20:47:27.885306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:27.885268 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 20:47:27.885883 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:27.885848 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 20:47:28.330563 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:28.330527 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 20:47:28.330714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:28.330577 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 20:47:28.871525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:28.871486 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 20:47:28.871803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:28.871776 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 20:47:29.330271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:29.330237 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 20:47:29.330670 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:29.330287 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 20:47:29.881180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:29.880658 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 20:47:29.881180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:29.880735 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 20:47:30.336128 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:30.336021 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 20:47:30.336894 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:30.336846 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 20:47:30.872180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:30.871492 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 20:47:30.872180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:30.871553 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 20:47:31.330063 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:31.330024 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 20:47:31.330256 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:31.330090 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 20:47:31.782637 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:31.782595 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:31.803029 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:31.802994 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:31.871581 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:31.871537 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 20:47:31.871795 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:31.871606 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 20:47:31.953525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:31.953486 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:47:33.084503 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:33.084466 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:33.086349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:33.086150 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:33.087373 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:33.087321 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:47:33.193292 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:33.193261 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:33.193604 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:33.193581 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:33.195077 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:47:33.195051 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 20:47:56.085018 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:47:56.084969 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:48:26.083470 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:48:26.083434 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:48:28.229303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.229261 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr"] Nov 03 20:48:28.233688 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.233667 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.303991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.303955 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-dshm\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.304173 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.304009 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-home\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.304173 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.304096 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26t6f\" (UniqueName: \"kubernetes.io/projected/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-kube-api-access-26t6f\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.304173 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.304137 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-model-cache\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.304351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.304179 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tmp-dir\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.304351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.304260 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tls-certs\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.329307 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.329279 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"openshift-service-ca.crt\"" Nov 03 20:48:28.329448 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.329337 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"distinct-w2-b119f2-kserve-self-signed-certs\"" Nov 03 20:48:28.329448 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.329405 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"default-dockercfg-bnmdp\"" Nov 03 20:48:28.329543 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.329521 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"kube-root-ca.crt\"" Nov 03 20:48:28.331746 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.331725 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr"] Nov 03 20:48:28.405192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.405128 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tmp-dir\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.405974 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.405946 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tls-certs\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.408314 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.406267 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-dshm\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.408509 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.408487 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-home\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.408695 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.408678 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26t6f\" (UniqueName: \"kubernetes.io/projected/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-kube-api-access-26t6f\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.409239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.409219 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-model-cache\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.413271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.409084 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-home\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.413271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.405873 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tmp-dir\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.413271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.409588 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-model-cache\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.418766 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.418518 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-dshm\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.420971 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.420950 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tls-certs\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.462285 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.462261 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26t6f\" (UniqueName: \"kubernetes.io/projected/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-kube-api-access-26t6f\") pod \"distinct-w2-b119f2-kserve-6dff96484d-ssfxr\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.545610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.545532 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:28.982381 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:28.982341 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr"] Nov 03 20:48:28.991151 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:48:28.991126 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a61d61f_f982_4f59_adaa_c0c7210bb7d3.slice/crio-e76d32f187410ae3916b1b2b2d96a7f21a488ae84da50167c7243cbf962ad4d1 WatchSource:0}: Error finding container e76d32f187410ae3916b1b2b2d96a7f21a488ae84da50167c7243cbf962ad4d1: Status 404 returned error can't find the container with id e76d32f187410ae3916b1b2b2d96a7f21a488ae84da50167c7243cbf962ad4d1 Nov 03 20:48:29.250053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:29.249948 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" event={"ID":"7a61d61f-f982-4f59-adaa-c0c7210bb7d3","Type":"ContainerStarted","Data":"e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592"} Nov 03 20:48:29.250053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:29.250008 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" event={"ID":"7a61d61f-f982-4f59-adaa-c0c7210bb7d3","Type":"ContainerStarted","Data":"e76d32f187410ae3916b1b2b2d96a7f21a488ae84da50167c7243cbf962ad4d1"} Nov 03 20:48:29.358204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:29.358142 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podStartSLOduration=1.358128224 podStartE2EDuration="1.358128224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:29.35607061 +0000 UTC m=+1176.593048495" watchObservedRunningTime="2026-11-03 20:48:29.358128224 +0000 UTC m=+1176.595106098" Nov 03 20:48:29.546272 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:29.546198 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:38.546037 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:38.546007 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:38.559147 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:38.559123 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:39.314770 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:39.314741 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:48:53.008776 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:53.008746 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:48:53.011468 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:48:53.011443 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:48:56.086279 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:48:56.086241 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:49:00.514117 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.514081 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq"] Nov 03 20:49:00.518298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.518281 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.614844 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.614811 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"openshift-service-ca.crt\"" Nov 03 20:49:00.614992 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.614928 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"distinct-w4-984cbc-kserve-self-signed-certs\"" Nov 03 20:49:00.615045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.614984 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"kube-root-ca.crt\"" Nov 03 20:49:00.615045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.614995 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"default-dockercfg-9fwld\"" Nov 03 20:49:00.617001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.616981 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq"] Nov 03 20:49:00.618431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.618409 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86280466-9e84-42a4-a0be-9798a081c9cd-tls-certs\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.618536 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.618445 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcn5\" (UniqueName: \"kubernetes.io/projected/86280466-9e84-42a4-a0be-9798a081c9cd-kube-api-access-lkcn5\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.618583 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.618544 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-home\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.618670 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.618655 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-tmp-dir\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.618707 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.618696 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-model-cache\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.618752 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.618712 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-dshm\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.730842 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86280466-9e84-42a4-a0be-9798a081c9cd-tls-certs\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.730907 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcn5\" (UniqueName: \"kubernetes.io/projected/86280466-9e84-42a4-a0be-9798a081c9cd-kube-api-access-lkcn5\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.730966 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-home\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.731082 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-tmp-dir\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.731131 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-model-cache\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.731171 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-dshm\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.732782 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-tmp-dir\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.733032 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-model-cache\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.733860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.733350 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-home\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.737245 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.737219 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-dshm\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.743750 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.743724 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86280466-9e84-42a4-a0be-9798a081c9cd-tls-certs\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.788836 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.788771 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcn5\" (UniqueName: \"kubernetes.io/projected/86280466-9e84-42a4-a0be-9798a081c9cd-kube-api-access-lkcn5\") pod \"distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:00.830457 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:00.830425 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:01.212484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:01.212452 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq"] Nov 03 20:49:01.219412 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:49:01.219381 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86280466_9e84_42a4_a0be_9798a081c9cd.slice/crio-df03275a3c3076ddf15a7f9805e2b6b9634afda23806bfb4ae33b7b26f0fda9d WatchSource:0}: Error finding container df03275a3c3076ddf15a7f9805e2b6b9634afda23806bfb4ae33b7b26f0fda9d: Status 404 returned error can't find the container with id df03275a3c3076ddf15a7f9805e2b6b9634afda23806bfb4ae33b7b26f0fda9d Nov 03 20:49:01.423788 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:01.423750 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" event={"ID":"86280466-9e84-42a4-a0be-9798a081c9cd","Type":"ContainerStarted","Data":"7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621"} Nov 03 20:49:01.423975 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:01.423797 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" event={"ID":"86280466-9e84-42a4-a0be-9798a081c9cd","Type":"ContainerStarted","Data":"df03275a3c3076ddf15a7f9805e2b6b9634afda23806bfb4ae33b7b26f0fda9d"} Nov 03 20:49:01.530853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:01.530761 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podStartSLOduration=1.530744718 podStartE2EDuration="1.530744718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:01.530238177 +0000 UTC m=+1208.767216045" watchObservedRunningTime="2026-11-03 20:49:01.530744718 +0000 UTC m=+1208.767722593" Nov 03 20:49:01.831534 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:01.831458 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:06.821829 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.821753 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f"] Nov 03 20:49:06.825630 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.825604 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.874362 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.874326 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fe2825-openshift-default-dockercfg-7zsdg\"" Nov 03 20:49:06.901246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901213 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901394 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901375 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901445 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901423 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93442648-c09f-4e6d-a521-9d4ed002812a-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901501 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901487 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901542 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901525 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901592 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901544 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgsm\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-kube-api-access-gqgsm\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901592 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901568 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901707 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901623 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.901764 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.901708 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93442648-c09f-4e6d-a521-9d4ed002812a-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:06.924706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:06.924673 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f"] Nov 03 20:49:07.002860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.002829 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.002865 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgsm\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-kube-api-access-gqgsm\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.002898 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.002922 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.002941 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93442648-c09f-4e6d-a521-9d4ed002812a-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.002988 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003114 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003209 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93442648-c09f-4e6d-a521-9d4ed002812a-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003256 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003402 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003521 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003621 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003662 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.003883 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.003863 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93442648-c09f-4e6d-a521-9d4ed002812a-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.005307 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.005285 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.005638 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.005615 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93442648-c09f-4e6d-a521-9d4ed002812a-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.055637 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.055608 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.055767 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.055749 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgsm\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-kube-api-access-gqgsm\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.141557 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.141490 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:07.571931 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.571896 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f"] Nov 03 20:49:07.578497 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:49:07.578465 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93442648_c09f_4e6d_a521_9d4ed002812a.slice/crio-b60d3b00f8a6b9bc42f56d010f657c7c98998ab2d45d5e76e88397df1d8d3b65 WatchSource:0}: Error finding container b60d3b00f8a6b9bc42f56d010f657c7c98998ab2d45d5e76e88397df1d8d3b65: Status 404 returned error can't find the container with id b60d3b00f8a6b9bc42f56d010f657c7c98998ab2d45d5e76e88397df1d8d3b65 Nov 03 20:49:07.581182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.581125 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:49:07.581272 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.581235 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:49:07.581322 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:07.581277 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:49:08.479541 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:08.479498 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" event={"ID":"93442648-c09f-4e6d-a521-9d4ed002812a","Type":"ContainerStarted","Data":"104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e"} Nov 03 20:49:08.480046 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:08.479566 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" event={"ID":"93442648-c09f-4e6d-a521-9d4ed002812a","Type":"ContainerStarted","Data":"b60d3b00f8a6b9bc42f56d010f657c7c98998ab2d45d5e76e88397df1d8d3b65"} Nov 03 20:49:08.614612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:08.614559 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podStartSLOduration=2.614543975 podStartE2EDuration="2.614543975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:08.61296997 +0000 UTC m=+1215.849947844" watchObservedRunningTime="2026-11-03 20:49:08.614543975 +0000 UTC m=+1215.851521849" Nov 03 20:49:09.141906 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:09.141873 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:09.143431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:09.143404 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:49:09.143578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:09.143468 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 20:49:10.142458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:10.142416 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:49:10.142839 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:10.142530 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 20:49:10.830865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:10.830825 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:10.844170 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:10.844134 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:11.142854 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:11.142773 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:49:11.142854 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:11.142831 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 20:49:11.508655 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:11.508628 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:49:12.142264 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:12.142228 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:49:12.142432 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:12.142303 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 20:49:13.142453 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:13.142409 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:49:13.142934 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:13.142469 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 20:49:14.142059 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:14.142021 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 20:49:14.142240 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:14.142098 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 20:49:15.146210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:15.146179 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:15.146695 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:15.146635 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:15.147367 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:49:15.147341 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 20:49:26.088730 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:49:26.088689 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:49:56.082438 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:49:56.082402 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:50:26.081221 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:50:26.081179 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:50:33.125872 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.125830 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l"] Nov 03 20:50:33.129828 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.129805 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.190667 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.190634 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-dshm\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.190835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.190719 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-model-cache\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.190835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.190817 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-tmp-dir\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.190958 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.190940 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c956f\" (UniqueName: \"kubernetes.io/projected/0352c748-5879-44cc-a69a-8fa2f1eedffa-kube-api-access-c956f\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.190998 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.190975 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-home\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.191073 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.191061 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352c748-5879-44cc-a69a-8fa2f1eedffa-tls-certs\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.225540 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.225504 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fe2825\"/\"distinct-2-w3-fe2825-kserve-self-signed-certs\"" Nov 03 20:50:33.228235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.228204 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l"] Nov 03 20:50:33.292603 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.292572 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-home\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.292780 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.292640 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352c748-5879-44cc-a69a-8fa2f1eedffa-tls-certs\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.292780 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.292726 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-dshm\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.292917 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.292824 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-model-cache\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.292917 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.292860 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-tmp-dir\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.293020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.292950 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c956f\" (UniqueName: \"kubernetes.io/projected/0352c748-5879-44cc-a69a-8fa2f1eedffa-kube-api-access-c956f\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.293075 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.293044 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-home\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.293206 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.293183 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-model-cache\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.293293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.293222 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-tmp-dir\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.295258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.295240 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352c748-5879-44cc-a69a-8fa2f1eedffa-tls-certs\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.295359 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.295339 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-dshm\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.347090 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.347054 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c956f\" (UniqueName: \"kubernetes.io/projected/0352c748-5879-44cc-a69a-8fa2f1eedffa-kube-api-access-c956f\") pod \"distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.446872 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.446837 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:33.959926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:33.959891 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l"] Nov 03 20:50:33.966522 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:50:33.966491 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0352c748_5879_44cc_a69a_8fa2f1eedffa.slice/crio-40f1888e1c933832c1d160e586a2d941058f919a3ea164f254df7aa8bebba870 WatchSource:0}: Error finding container 40f1888e1c933832c1d160e586a2d941058f919a3ea164f254df7aa8bebba870: Status 404 returned error can't find the container with id 40f1888e1c933832c1d160e586a2d941058f919a3ea164f254df7aa8bebba870 Nov 03 20:50:34.962677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:34.962644 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" event={"ID":"0352c748-5879-44cc-a69a-8fa2f1eedffa","Type":"ContainerStarted","Data":"e34495afed11661bd43e64b11f49ef924462471c97622c6c10bafba9ba4d9538"} Nov 03 20:50:34.963197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:34.962685 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" event={"ID":"0352c748-5879-44cc-a69a-8fa2f1eedffa","Type":"ContainerStarted","Data":"40f1888e1c933832c1d160e586a2d941058f919a3ea164f254df7aa8bebba870"} Nov 03 20:50:35.069805 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:35.069746 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podStartSLOduration=2.069727956 podStartE2EDuration="2.069727956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:35.067875151 +0000 UTC m=+1302.304853024" watchObservedRunningTime="2026-11-03 20:50:35.069727956 +0000 UTC m=+1302.306705831" Nov 03 20:50:35.447898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:35.447860 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:41.523017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.522984 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48"] Nov 03 20:50:41.527419 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.527391 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.641993 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.641955 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.642370 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.642341 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.642617 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.642592 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b082e35-1549-4590-93f3-2713bfdda109-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.642830 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.642809 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.643139 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.643122 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgsxd\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-kube-api-access-kgsxd\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.643320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.643306 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.646142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.646118 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.646389 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.646363 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b082e35-1549-4590-93f3-2713bfdda109-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.646521 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.646495 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.660428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.660391 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48"] Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.750948 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751018 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b082e35-1549-4590-93f3-2713bfdda109-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751092 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751135 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751181 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751220 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b082e35-1549-4590-93f3-2713bfdda109-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751251 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751369 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgsxd\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-kube-api-access-kgsxd\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751398 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.751862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.752881 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b082e35-1549-4590-93f3-2713bfdda109-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.753187 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.753455 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.758192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.753918 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.804755 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.802103 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b082e35-1549-4590-93f3-2713bfdda109-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.805550 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.805521 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.812449 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.812425 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.848031 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.848001 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgsxd\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-kube-api-access-kgsxd\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:41.861092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:41.861061 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:42.290931 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:42.290891 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48"] Nov 03 20:50:42.300380 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:50:42.299555 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b082e35_1549_4590_93f3_2713bfdda109.slice/crio-d7d1fdc629481d4608877613c944d9bdf7ab1ac3d2a64e0dbb897a20186a8cee WatchSource:0}: Error finding container d7d1fdc629481d4608877613c944d9bdf7ab1ac3d2a64e0dbb897a20186a8cee: Status 404 returned error can't find the container with id d7d1fdc629481d4608877613c944d9bdf7ab1ac3d2a64e0dbb897a20186a8cee Nov 03 20:50:42.307131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:42.307075 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:50:42.309294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:42.309261 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:50:42.309560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:42.309535 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:50:43.011093 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.011059 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" event={"ID":"6b082e35-1549-4590-93f3-2713bfdda109","Type":"ContainerStarted","Data":"24cd5f0f735798be57f3250ba476b76fafb58c3c231fe1f6a9429a1e071acf19"} Nov 03 20:50:43.011519 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.011117 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" event={"ID":"6b082e35-1549-4590-93f3-2713bfdda109","Type":"ContainerStarted","Data":"d7d1fdc629481d4608877613c944d9bdf7ab1ac3d2a64e0dbb897a20186a8cee"} Nov 03 20:50:43.123057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.122991 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podStartSLOduration=2.122976093 podStartE2EDuration="2.122976093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:43.121390876 +0000 UTC m=+1310.358368750" watchObservedRunningTime="2026-11-03 20:50:43.122976093 +0000 UTC m=+1310.359953968" Nov 03 20:50:43.452181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.451489 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:43.491672 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.491639 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:43.863241 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.862231 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:43.863441 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.863342 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:50:43.863441 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:43.863407 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 20:50:44.029356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:44.029324 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:50:44.865375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:44.865337 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:50:44.865668 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:44.865639 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 20:50:45.862185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:45.861533 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:50:45.862185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:45.861593 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 20:50:46.743708 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.743674 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq"] Nov 03 20:50:46.767183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.751552 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.802991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.802941 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq"] Nov 03 20:50:46.822060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.822020 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-home\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.822229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.822074 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-dshm\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.822298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.822280 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-tmp-dir\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.822353 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.822334 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7pw\" (UniqueName: \"kubernetes.io/projected/05139074-b3a2-4cdc-8041-32fd809250ac-kube-api-access-wn7pw\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.822446 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.822422 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05139074-b3a2-4cdc-8041-32fd809250ac-tls-certs\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.822629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.822602 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-model-cache\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.843759 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.843730 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"distinct-2-w0-fd9afe-kserve-self-signed-certs\"" Nov 03 20:50:46.877458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.877336 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:50:46.877458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.877405 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.951774 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-model-cache\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.951899 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-home\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.951928 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-dshm\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.951985 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-tmp-dir\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.952024 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7pw\" (UniqueName: \"kubernetes.io/projected/05139074-b3a2-4cdc-8041-32fd809250ac-kube-api-access-wn7pw\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.952056 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05139074-b3a2-4cdc-8041-32fd809250ac-tls-certs\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.952596 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-model-cache\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.952838 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-home\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.959186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.955862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-tmp-dir\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.991761 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.991729 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05139074-b3a2-4cdc-8041-32fd809250ac-tls-certs\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:46.997235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:46.993400 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-dshm\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:47.014213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:47.012944 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7pw\" (UniqueName: \"kubernetes.io/projected/05139074-b3a2-4cdc-8041-32fd809250ac-kube-api-access-wn7pw\") pod \"distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:47.088181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:47.087522 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:47.701195 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:47.701146 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq"] Nov 03 20:50:47.706624 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:50:47.706594 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05139074_b3a2_4cdc_8041_32fd809250ac.slice/crio-48381f79a1e81ae45f884be580f5feba2a2b7903730cc21b4f18caef1414a688 WatchSource:0}: Error finding container 48381f79a1e81ae45f884be580f5feba2a2b7903730cc21b4f18caef1414a688: Status 404 returned error can't find the container with id 48381f79a1e81ae45f884be580f5feba2a2b7903730cc21b4f18caef1414a688 Nov 03 20:50:47.871186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:47.865342 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:50:47.871186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:47.865409 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 20:50:48.041565 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:48.041489 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" event={"ID":"05139074-b3a2-4cdc-8041-32fd809250ac","Type":"ContainerStarted","Data":"739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423"} Nov 03 20:50:48.041565 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:48.041530 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" event={"ID":"05139074-b3a2-4cdc-8041-32fd809250ac","Type":"ContainerStarted","Data":"48381f79a1e81ae45f884be580f5feba2a2b7903730cc21b4f18caef1414a688"} Nov 03 20:50:48.089652 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:48.088460 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:48.179912 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:48.179851 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podStartSLOduration=2.179837415 podStartE2EDuration="2.179837415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:48.177596554 +0000 UTC m=+1315.414574430" watchObservedRunningTime="2026-11-03 20:50:48.179837415 +0000 UTC m=+1315.416815288" Nov 03 20:50:48.862368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:48.862335 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:50:48.862564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:48.862389 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 20:50:49.865885 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:49.865855 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:49.866349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:49.866336 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:49.867095 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:49.867071 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 20:50:56.081216 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:50:56.081183 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:50:57.087950 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:57.087919 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:57.100815 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:57.100786 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:50:58.100453 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:50:58.100420 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:51:26.087472 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:51:26.087199 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:51:26.852462 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.852420 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm"] Nov 03 20:51:26.857204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.857147 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.947407 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.947365 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rh62\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-kube-api-access-7rh62\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.947578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.947420 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.947578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.947449 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.947578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.947481 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.947578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.947559 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.947796 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.947593 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.948035 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.948003 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.948123 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.948040 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e756a29-2f53-45a6-910c-e9b5907235d8-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:26.948123 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:26.948100 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.049580 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.049535 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm"] Nov 03 20:51:27.049858 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.049827 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rh62\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-kube-api-access-7rh62\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.050243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.050218 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.050349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.050261 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.050349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.050319 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.050458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.050401 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.051464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.050431 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.051464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.050970 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.051464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.051043 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e756a29-2f53-45a6-910c-e9b5907235d8-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.051464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.051089 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.051464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.051099 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.051464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.051359 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.051814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.051601 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.052426 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.052391 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.053001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.052977 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e756a29-2f53-45a6-910c-e9b5907235d8-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.058736 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.058692 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.060075 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.060050 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.103627 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.103568 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rh62\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-kube-api-access-7rh62\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.103920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.103769 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.175257 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.175213 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:27.599214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.599180 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm"] Nov 03 20:51:27.605440 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:51:27.605413 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e756a29_2f53_45a6_910c_e9b5907235d8.slice/crio-da17e961bf88f6cd7a577502b1cbebd365643f9665186134443c3b5fcd770ee3 WatchSource:0}: Error finding container da17e961bf88f6cd7a577502b1cbebd365643f9665186134443c3b5fcd770ee3: Status 404 returned error can't find the container with id da17e961bf88f6cd7a577502b1cbebd365643f9665186134443c3b5fcd770ee3 Nov 03 20:51:27.607667 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.607636 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:27.607747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.607716 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:27.607785 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:27.607745 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:28.399579 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:28.399535 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" event={"ID":"1e756a29-2f53-45a6-910c-e9b5907235d8","Type":"ContainerStarted","Data":"3deb5f5a2782f1a988cf54e2b22e6db4f91f00defd347216bfe0f2721d596683"} Nov 03 20:51:28.400445 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:28.399604 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" event={"ID":"1e756a29-2f53-45a6-910c-e9b5907235d8","Type":"ContainerStarted","Data":"da17e961bf88f6cd7a577502b1cbebd365643f9665186134443c3b5fcd770ee3"} Nov 03 20:51:28.612202 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:28.612134 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podStartSLOduration=2.612119982 podStartE2EDuration="2.612119982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.564717868 +0000 UTC m=+1355.801695747" watchObservedRunningTime="2026-11-03 20:51:28.612119982 +0000 UTC m=+1355.849097856" Nov 03 20:51:29.177776 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:29.177744 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:29.178136 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:29.178109 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:51:29.178353 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:29.178326 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:51:30.190633 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:30.190598 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:51:30.191309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:30.191206 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:51:31.181178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:31.180598 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:51:31.181178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:31.180690 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:51:31.549235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:31.548305 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:31.998184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:31.980256 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.042381 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.042329 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.044658 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.044604 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.316012 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.315926 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:51:32.489423 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.489372 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:51:32.507182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.319309 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.551308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.551271 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 20:51:32.552530 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.552499 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.562179 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.558438 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:51:32.718506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.718466 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 20:51:32.769338 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.769287 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:32.787263 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.787222 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Nov 03 20:51:32.875313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:32.875249 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 20:51:33.136984 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:33.136871 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:33.152180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:33.145051 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:33.184564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:33.184529 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:51:33.184829 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:33.184792 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:51:33.271182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:33.259696 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:51:34.178931 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:34.178894 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:51:34.179620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:34.179558 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:51:35.176601 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:35.176554 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:51:35.176818 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:35.176619 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:51:37.218297 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:37.218260 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:51:37.218782 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:37.218322 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:37.393287 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:37.393252 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:37.393551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:37.393529 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:37.394557 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:37.394534 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:51:39.659351 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.659306 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg"] Nov 03 20:51:39.663630 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.663605 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.708815 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.708787 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtzt7\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-kube-api-access-mtzt7\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.708937 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.708862 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88970ce2-dfab-4462-b5d9-3797ea5d2072-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.708937 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.708898 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.709010 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.709001 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.709048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.709016 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.709126 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.709106 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.709182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.709142 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.709223 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.709181 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.709223 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.709202 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.765116 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.765076 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg"] Nov 03 20:51:39.810189 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810128 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810372 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810196 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810372 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810253 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810372 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810280 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810372 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810314 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810372 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810341 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810678 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810398 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtzt7\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-kube-api-access-mtzt7\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810678 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810498 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88970ce2-dfab-4462-b5d9-3797ea5d2072-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810678 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810562 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.810845 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.810794 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.811198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.811149 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.811668 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.811557 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.811668 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.811624 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88970ce2-dfab-4462-b5d9-3797ea5d2072-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.811792 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.811693 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.813081 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.813063 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.813183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.813099 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.863098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.863071 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.864236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.864212 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtzt7\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-kube-api-access-mtzt7\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:39.979289 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:39.979201 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:40.403894 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.403863 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg"] Nov 03 20:51:40.410931 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:51:40.410902 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88970ce2_dfab_4462_b5d9_3797ea5d2072.slice/crio-83716b4583b366a763c77f71804565c28ec15f2adfc28f7112bd3709ebe29100 WatchSource:0}: Error finding container 83716b4583b366a763c77f71804565c28ec15f2adfc28f7112bd3709ebe29100: Status 404 returned error can't find the container with id 83716b4583b366a763c77f71804565c28ec15f2adfc28f7112bd3709ebe29100 Nov 03 20:51:40.413364 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.413333 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:40.413465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.413405 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:40.413465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.413450 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:40.565798 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.565760 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" event={"ID":"88970ce2-dfab-4462-b5d9-3797ea5d2072","Type":"ContainerStarted","Data":"56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b"} Nov 03 20:51:40.566002 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.565823 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" event={"ID":"88970ce2-dfab-4462-b5d9-3797ea5d2072","Type":"ContainerStarted","Data":"83716b4583b366a763c77f71804565c28ec15f2adfc28f7112bd3709ebe29100"} Nov 03 20:51:40.675789 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.675685 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podStartSLOduration=1.6756655550000001 podStartE2EDuration="1.675665555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:40.673505719 +0000 UTC m=+1367.910483606" watchObservedRunningTime="2026-11-03 20:51:40.675665555 +0000 UTC m=+1367.912643432" Nov 03 20:51:40.983238 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:40.982268 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:41.941943 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:41.941914 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg"] Nov 03 20:51:41.945565 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:41.945547 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:41.981619 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:41.981591 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:51:41.981751 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:41.981646 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:51:42.036341 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.036309 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg"] Nov 03 20:51:42.072116 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.071951 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bznt\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-kube-api-access-4bznt\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072116 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072008 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072179 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072347 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072417 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072505 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072534 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072579 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.072714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.072661 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174036 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174004 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174043 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174120 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174147 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174203 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174276 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174331 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bznt\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-kube-api-access-4bznt\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174366 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174410 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174655 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.174722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.174701 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.175435 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.175368 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.175435 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.175371 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.175938 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.175917 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.177464 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.177436 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.177673 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.177657 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.227927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.227864 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.228215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.228193 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bznt\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-kube-api-access-4bznt\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.275524 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.275493 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:42.702361 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.702329 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg"] Nov 03 20:51:42.774044 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:51:42.774002 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402854d1_16ef_4ed1_9b1f_c6643af60b3b.slice/crio-e219964023eb57111dc819e772f3295c623fdd3d90bfe2240abbc28457427f20 WatchSource:0}: Error finding container e219964023eb57111dc819e772f3295c623fdd3d90bfe2240abbc28457427f20: Status 404 returned error can't find the container with id e219964023eb57111dc819e772f3295c623fdd3d90bfe2240abbc28457427f20 Nov 03 20:51:42.776853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.776814 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:42.776954 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.776904 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:42.776954 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.776945 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:42.980076 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.979998 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:51:42.980076 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:42.980047 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:51:43.611188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:43.600559 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" event={"ID":"402854d1-16ef-4ed1-9b1f-c6643af60b3b","Type":"ContainerStarted","Data":"5362962e9f11fb11fb9e606c01e7a8d73c7f70385ce01e025e8cf379992e2dd6"} Nov 03 20:51:43.611188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:43.600631 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" event={"ID":"402854d1-16ef-4ed1-9b1f-c6643af60b3b","Type":"ContainerStarted","Data":"e219964023eb57111dc819e772f3295c623fdd3d90bfe2240abbc28457427f20"} Nov 03 20:51:43.988396 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:43.988356 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:51:43.989057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:43.989004 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:51:44.279726 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:44.279635 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:44.280109 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:44.280085 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:51:44.280306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:44.280280 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:51:44.983565 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:44.983523 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:51:44.983852 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:44.983822 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:51:45.278237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:45.277453 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:51:45.278237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:45.277512 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:51:45.985546 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:45.985511 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:51:45.985781 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:45.985757 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:51:46.279951 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:46.279487 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:51:46.279951 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:46.279542 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:51:46.979657 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:46.979624 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:51:46.979838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:46.979674 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:51:47.278238 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:47.277785 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:51:47.278238 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:47.277845 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:51:47.984354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:47.984323 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:47.984805 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:47.984744 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:47.985441 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:47.985425 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 20:51:48.141171 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:48.141099 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podStartSLOduration=7.141085073 podStartE2EDuration="7.141085073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:43.809508286 +0000 UTC m=+1371.046486161" watchObservedRunningTime="2026-11-03 20:51:48.141085073 +0000 UTC m=+1375.378062946" Nov 03 20:51:48.278893 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:48.278809 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:51:48.279111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:48.279083 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:51:49.276984 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:49.276933 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:51:49.277409 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:49.276998 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:51:50.280044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:50.280013 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:50.280477 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:50.280364 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:50.281068 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:50.281048 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:51:54.765044 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.765003 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4"] Nov 03 20:51:54.769458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.769432 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.811091 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.811064 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7"] Nov 03 20:51:54.814919 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.814903 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.859070 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.859039 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4"] Nov 03 20:51:54.879023 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.878998 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7"] Nov 03 20:51:54.887802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.887770 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36240b08-909a-47c5-a475-8d8fac2fb203-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.887926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.887818 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.887926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.887848 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.887926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.887871 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.888054 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888002 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888099 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888061 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.888099 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888087 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888126 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.888191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888151 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.888276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888232 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36240b08-909a-47c5-a475-8d8fac2fb203-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.888276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888260 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888346 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888293 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sdsm\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-kube-api-access-7sdsm\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.888383 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888370 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888453 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888437 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snbn7\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-kube-api-access-snbn7\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888495 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888469 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888495 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888487 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888608 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888589 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.888670 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.888651 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.989655 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989625 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snbn7\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-kube-api-access-snbn7\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.989825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989664 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.989825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989682 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.989825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989701 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.989825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989745 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.989825 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989805 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36240b08-909a-47c5-a475-8d8fac2fb203-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989830 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989858 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989883 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989942 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.989997 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990022 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990060 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990082 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990128 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36240b08-909a-47c5-a475-8d8fac2fb203-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990154 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990204 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sdsm\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-kube-api-access-7sdsm\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990207 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990228 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990277 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.990505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990320 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990872 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990531 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990872 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990671 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.990992 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.990947 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.991113 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.991086 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36240b08-909a-47c5-a475-8d8fac2fb203-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.991221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.991200 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.991624 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.991598 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.992205 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.991805 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:54.993569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.993541 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36240b08-909a-47c5-a475-8d8fac2fb203-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.993674 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.993653 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:54.996261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:54.994214 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:55.004913 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.004725 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:55.043538 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.043478 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:55.044107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.044080 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:55.044360 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.044342 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snbn7\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-kube-api-access-snbn7\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:55.044417 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.044351 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sdsm\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-kube-api-access-7sdsm\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:55.083836 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.083816 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:55.128904 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.128868 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:55.525263 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.525069 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4"] Nov 03 20:51:55.583056 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.581284 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7"] Nov 03 20:51:55.588550 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.587938 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:55.588550 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.588028 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:55.588550 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.588069 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:55.596598 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:51:55.596571 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef43ca2_319b_4b4a_bbc4_4bde9da7c1ed.slice/crio-24b3621c5ff5ad1a22fb3813603e45443ee51e5eb9403afc73ee9438100b0f55 WatchSource:0}: Error finding container 24b3621c5ff5ad1a22fb3813603e45443ee51e5eb9403afc73ee9438100b0f55: Status 404 returned error can't find the container with id 24b3621c5ff5ad1a22fb3813603e45443ee51e5eb9403afc73ee9438100b0f55 Nov 03 20:51:55.603538 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.603503 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:55.603634 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.603588 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:55.603678 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.603630 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:51:55.715895 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.714466 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" event={"ID":"36240b08-909a-47c5-a475-8d8fac2fb203","Type":"ContainerStarted","Data":"3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581"} Nov 03 20:51:55.715895 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.714530 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" event={"ID":"36240b08-909a-47c5-a475-8d8fac2fb203","Type":"ContainerStarted","Data":"f29c12455a889e75f61011b6086d0b0f38f8c454b1a5b292ac96c876c6a8b252"} Nov 03 20:51:55.716503 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.716474 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" event={"ID":"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed","Type":"ContainerStarted","Data":"f46b9b33f6dbe12f1db92082d86c409a1f87ea07835731e0e9f6806747b4c417"} Nov 03 20:51:55.716786 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.716764 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" event={"ID":"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed","Type":"ContainerStarted","Data":"24b3621c5ff5ad1a22fb3813603e45443ee51e5eb9403afc73ee9438100b0f55"} Nov 03 20:51:55.828100 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.828005 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podStartSLOduration=1.827991299 podStartE2EDuration="1.827991299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:55.825437543 +0000 UTC m=+1383.062415418" watchObservedRunningTime="2026-11-03 20:51:55.827991299 +0000 UTC m=+1383.064969172" Nov 03 20:51:55.937274 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:55.937219 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podStartSLOduration=1.9372035250000001 podStartE2EDuration="1.937203525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:55.933964457 +0000 UTC m=+1383.170942333" watchObservedRunningTime="2026-11-03 20:51:55.937203525 +0000 UTC m=+1383.174181399" Nov 03 20:51:56.084582 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:56.084501 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:51:56.088331 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:51:56.088210 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:51:56.089001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:56.088721 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:51:56.089001 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:56.088781 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:51:56.129889 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:56.129861 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:51:56.131022 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:56.131001 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:51:56.131075 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:56.131053 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:51:57.084764 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:57.084726 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:51:57.085440 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:57.085394 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:51:57.129544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:57.129507 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:51:57.129729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:57.129567 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:51:58.089562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:58.089521 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:51:58.090215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:58.090153 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:51:58.129460 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:58.129433 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:51:58.129662 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:58.129638 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:51:59.092183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:59.088446 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:51:59.092183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:59.088505 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:51:59.129315 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:59.129279 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:51:59.129475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:51:59.129333 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:52:00.087593 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:00.087555 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:52:00.087889 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:00.087861 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:52:00.130224 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:00.130195 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:52:00.130665 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:00.130637 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:52:01.085183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:01.084492 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:52:01.085183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:01.084551 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:52:01.131814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:01.131764 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:52:01.133048 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:01.133002 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:52:02.084971 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:02.084927 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:52:02.085186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:02.085002 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:52:02.129213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:02.129174 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:52:02.129379 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:02.129238 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:52:03.244553 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:03.244506 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:52:03.245041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:03.244937 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:52:03.245905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:03.245882 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:52:03.296252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:03.296225 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:52:03.296679 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:03.296647 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:52:03.297676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:03.297659 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 20:52:06.955207 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:06.943582 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:06.998181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:06.993345 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Nov 03 20:52:07.072364 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.072289 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.167489 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.167441 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.168921 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.168790 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.361265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.360326 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 20:52:07.361265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.360851 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.548364 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.548319 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.768310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.768259 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.834295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.834244 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:07.951203 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.949405 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 20:52:07.952186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:07.952051 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:52:08.102236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.094783 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:52:08.102236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.095592 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:52:08.102236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.097416 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.179251 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.173735 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:08.199216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.186359 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 20:52:08.199216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.187230 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:52:08.199216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.187312 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:52:08.275214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.253733 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.427183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.422616 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.427183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.423037 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 20:52:08.427183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.423470 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:52:08.427183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.423554 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:52:08.544219 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.463690 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:08.544219 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.463749 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.557265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.557221 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.589184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.563661 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:52:08.778239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.757834 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.779958 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.779910 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:08.784366 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.784345 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:52:08.897182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:08.896390 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 20:52:09.028185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.021301 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 20:52:09.097342 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.097233 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.193333 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.193289 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.299906 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.297102 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 20:52:09.442239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.381729 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:52:09.442581 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.442551 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:52:09.544604 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.544557 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.546922 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.546891 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 20:52:09.557626 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.557595 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 03 20:52:09.586729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.586691 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.637600 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.637564 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:52:09.760457 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.760349 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:09.773591 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.773561 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:52:09.787193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.784227 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 20:52:09.836715 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.836686 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:52:09.927772 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:09.927307 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.095236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.057717 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:52:10.132566 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.132521 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.274299 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.274251 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.360250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.360083 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:52:10.560273 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.545235 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.659172 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.560553 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 20:52:10.659172 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.595577 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:10.825328 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:10.825286 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.036022 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.035013 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.066697 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.066653 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 20:52:11.188240 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.142105 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.339816 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.339726 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.503127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.503091 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:52:11.507653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.507631 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:52:11.566259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.566210 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.592620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.592520 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:52:11.593973 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.593952 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:52:11.605569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.605201 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:11.627802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:11.627778 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:52:25.050121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.050087 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c"] Nov 03 20:52:25.054256 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.054235 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.106622 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.106594 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.106756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.106632 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.106756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.106657 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.106756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.106712 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.106874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.106827 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.106874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.106848 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.106956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.106867 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.107026 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.107014 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.107426 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.107405 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-kube-api-access-lgs5p\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.152932 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.152895 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c"] Nov 03 20:52:25.208908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.208876 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.208921 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.208959 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209067 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209138 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209190 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209222 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209346 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209452 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-kube-api-access-lgs5p\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209955 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209805 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209955 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209814 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.209955 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.209878 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.210222 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.210032 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.210310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.210289 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.213119 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.213090 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.213375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.213351 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.266093 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.266051 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.266519 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.266470 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-kube-api-access-lgs5p\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.369496 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.369425 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:25.902051 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.902021 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c"] Nov 03 20:52:25.908584 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:52:25.908555 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93102cbf_09e6_4941_b8e6_d59cc01d8bee.slice/crio-62c89f9fdfbd88149c793f218d4788d6382895af53fae8a8e9075f8e157803f2 WatchSource:0}: Error finding container 62c89f9fdfbd88149c793f218d4788d6382895af53fae8a8e9075f8e157803f2: Status 404 returned error can't find the container with id 62c89f9fdfbd88149c793f218d4788d6382895af53fae8a8e9075f8e157803f2 Nov 03 20:52:25.911281 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.911212 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:25.911400 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.911331 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:25.911400 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:25.911380 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:26.098974 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:52:26.098937 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:52:26.600434 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:26.600384 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" event={"ID":"93102cbf-09e6-4941-b8e6-d59cc01d8bee","Type":"ContainerStarted","Data":"4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65"} Nov 03 20:52:26.600607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:26.600457 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" event={"ID":"93102cbf-09e6-4941-b8e6-d59cc01d8bee","Type":"ContainerStarted","Data":"62c89f9fdfbd88149c793f218d4788d6382895af53fae8a8e9075f8e157803f2"} Nov 03 20:52:27.291849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.291799 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podStartSLOduration=2.291782771 podStartE2EDuration="2.291782771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:26.777242854 +0000 UTC m=+1414.014220728" watchObservedRunningTime="2026-11-03 20:52:27.291782771 +0000 UTC m=+1414.528760645" Nov 03 20:52:27.292492 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.292474 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb"] Nov 03 20:52:27.296405 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.296377 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.339228 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.339205 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6"] Nov 03 20:52:27.389299 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.389264 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:52:27.389503 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.389323 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 20:52:27.392416 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.392378 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:27.392535 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.392500 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.393469 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.393396 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb"] Nov 03 20:52:27.397692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.397666 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6"] Nov 03 20:52:27.491545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491503 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.491545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491549 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdxs\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-kube-api-access-lrdxs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.491771 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491632 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.491771 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491750 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.491842 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491790 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.491842 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491815 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.491924 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491866 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed04739-53d3-4918-9aff-21d4e40945eb-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.491960 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491921 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.492004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491962 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.492004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.491996 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.492088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492012 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.492088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492033 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed04739-53d3-4918-9aff-21d4e40945eb-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.492088 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492081 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7z9h\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-kube-api-access-g7z9h\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.492214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492153 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.492214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492207 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.492303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492222 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.492375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492357 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.492452 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.492435 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593228 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593130 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593228 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593187 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.593228 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593213 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed04739-53d3-4918-9aff-21d4e40945eb-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593228 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593230 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593251 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593293 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593318 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593348 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed04739-53d3-4918-9aff-21d4e40945eb-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593385 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7z9h\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-kube-api-access-g7z9h\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593451 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593482 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593508 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.593560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593554 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593567 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.594045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593583 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.594045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593731 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593871 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.594045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593888 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.593967 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.594045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594024 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed04739-53d3-4918-9aff-21d4e40945eb-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594055 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594090 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594092 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594139 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdxs\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-kube-api-access-lrdxs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594207 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594250 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594293 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.594431 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.594359 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.596378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.596350 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.596530 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.596510 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.596600 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.596578 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.596692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.596679 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed04739-53d3-4918-9aff-21d4e40945eb-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.669261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.668916 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdxs\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-kube-api-access-lrdxs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.673456 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.673426 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7z9h\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-kube-api-access-g7z9h\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.674031 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.674003 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.674451 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.674435 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:27.711449 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.711412 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:27.910976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:27.910953 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:28.175057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.174772 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6"] Nov 03 20:52:28.185038 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.185003 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:28.185193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.185088 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:28.185193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.185133 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:28.321989 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.321937 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb"] Nov 03 20:52:28.327070 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:52:28.327040 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f4bbf3_cfda_43a4_bf24_359a4bca5dcc.slice/crio-7a8f34e098daa161f3720e454befa10687c669d800a52380b465e85e18900b54 WatchSource:0}: Error finding container 7a8f34e098daa161f3720e454befa10687c669d800a52380b465e85e18900b54: Status 404 returned error can't find the container with id 7a8f34e098daa161f3720e454befa10687c669d800a52380b465e85e18900b54 Nov 03 20:52:28.329675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.329639 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:28.329779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.329747 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:28.329844 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.329793 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:28.377336 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.377307 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:52:28.377527 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.377502 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 20:52:28.623698 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.620330 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" event={"ID":"fed04739-53d3-4918-9aff-21d4e40945eb","Type":"ContainerStarted","Data":"ddf2638fd38ede1682a6359028be9dad296db3dda8c7a7246735883b8d515728"} Nov 03 20:52:28.623698 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.620413 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" event={"ID":"fed04739-53d3-4918-9aff-21d4e40945eb","Type":"ContainerStarted","Data":"21f1659824ad2e7f18da4400f0fd051af4e7e818f1028077969f8a2db69aa13a"} Nov 03 20:52:28.623698 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.622736 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" event={"ID":"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc","Type":"ContainerStarted","Data":"1a33dd01f50d16647487b9cbb673725c2304021151edf078c42242042d13a93b"} Nov 03 20:52:28.623698 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.622770 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" event={"ID":"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc","Type":"ContainerStarted","Data":"7a8f34e098daa161f3720e454befa10687c669d800a52380b465e85e18900b54"} Nov 03 20:52:28.712246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.712207 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:28.731269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.731199 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podStartSLOduration=1.731181468 podStartE2EDuration="1.731181468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:28.728541263 +0000 UTC m=+1415.965519154" watchObservedRunningTime="2026-11-03 20:52:28.731181468 +0000 UTC m=+1415.968159342" Nov 03 20:52:28.842792 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.842741 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podStartSLOduration=1.84272649 podStartE2EDuration="1.84272649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:28.839133982 +0000 UTC m=+1416.076111867" watchObservedRunningTime="2026-11-03 20:52:28.84272649 +0000 UTC m=+1416.079704364" Nov 03 20:52:28.911281 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:28.911243 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:29.371240 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:29.370520 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.371240 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:29.370581 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 20:52:29.720091 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:29.720056 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.720437 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:29.720394 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 20:52:29.915183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:29.914509 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.915183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:29.914591 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:52:30.373705 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:30.373252 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.373705 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:30.373314 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 20:52:30.721246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:30.715387 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.721246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:30.715450 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 20:52:30.914564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:30.914526 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.914839 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:30.914807 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:52:31.374186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:31.370823 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.374186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:31.370882 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 20:52:31.716104 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:31.716064 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.716385 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:31.716351 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 20:52:31.919519 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:31.919481 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.919780 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:31.919751 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:52:32.372183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:32.371435 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.372183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:32.371492 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 20:52:32.725554 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:32.725519 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.726194 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:32.726129 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 20:52:32.927937 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:32.927676 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.927937 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:32.927733 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:52:33.726122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:33.725187 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:52:33.726122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:33.725250 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 20:52:33.940575 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:33.940539 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:52:33.940819 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:33.940794 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:52:34.335618 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:34.335584 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:34.340246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:34.338342 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:34.340246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:34.340153 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 20:52:34.713375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:34.713340 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:52:34.713569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:34.713417 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 20:52:34.912446 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:34.912417 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:52:34.912844 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:34.912465 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:52:36.319313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:36.319280 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:36.320571 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:36.320527 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:36.321741 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:36.321717 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:52:36.441861 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:36.441827 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:36.442388 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:36.442365 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:36.450990 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:36.444422 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:52:38.204087 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:38.204052 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b"] Nov 03 20:52:38.204653 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:38.204273 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" containerID="cri-o://00ce6e13a42bdec6b4fddf7fcab2c277381377591f4e755c898447eb83461b34" gracePeriod=30 Nov 03 20:52:43.906408 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:43.906362 4227 generic.go:356] "Generic (PLEG): container finished" podID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerID="00ce6e13a42bdec6b4fddf7fcab2c277381377591f4e755c898447eb83461b34" exitCode=0 Nov 03 20:52:43.917309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:43.907123 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" event={"ID":"b3be870e-37e3-4a37-80cd-f4a2642ee303","Type":"ContainerDied","Data":"00ce6e13a42bdec6b4fddf7fcab2c277381377591f4e755c898447eb83461b34"} Nov 03 20:52:44.027902 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.027833 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.078587 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.078543 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.363261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.270612 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.363261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.322283 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.363261 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.322694 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.380245 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.368321 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.632273 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.595873 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 20:52:44.632273 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.598821 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 20:52:44.632273 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.598911 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:44.632574 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.632342 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 20:52:44.834387 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:44.833183 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.030297 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.030243 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.083426 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.083370 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.108615 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.108582 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:52:45.115553 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.115524 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:52:45.137611 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.137578 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:52:45.287341 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.287244 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:45.290004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.289981 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:52:45.291257 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.287915 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-kube-api-access-m95wj\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-kube-api-access-m95wj\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.291883 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.291866 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-podinfo\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.297279 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.297259 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-token\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-token\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.297755 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.297736 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-envoy\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.298321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.298300 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-data\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.314105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.314084 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-certs\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.319655 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.319636 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-credential-socket\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.322386 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.322366 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3be870e-37e3-4a37-80cd-f4a2642ee303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be870e-37e3-4a37-80cd-f4a2642ee303-istiod-ca-cert\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.331683 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.331661 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-socket\") pod \"b3be870e-37e3-4a37-80cd-f4a2642ee303\" (UID: \"b3be870e-37e3-4a37-80cd-f4a2642ee303\") " Nov 03 20:52:45.333218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.322560 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.345197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.331335 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.345197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.312750 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-data" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:45.345197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.319538 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-certs" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:45.345197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.322240 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-credential-socket" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:45.345197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.331533 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3be870e-37e3-4a37-80cd-f4a2642ee303-istiod-ca-cert" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:45.345197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.332053 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-socket" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:45.345956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.345936 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:52:45.346101 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.346091 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:52:45.393259 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.391975 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.397693 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.397287 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:52:45.485194 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.484753 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-envoy" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:45.553237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.518520 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-kube-api-access-m95wj" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "kube-api-access-m95wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:45.604053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.596745 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-token" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:45.604053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.596832 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.604053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.597278 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:52:45.604053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.603233 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.604053 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.603585 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:52:45.647185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.637239 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:45.647185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.637660 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:52:45.668590 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.668553 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-podinfo" pod "b3be870e-37e3-4a37-80cd-f4a2642ee303" (UID: "b3be870e-37e3-4a37-80cd-f4a2642ee303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:45.791185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.682599 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.849646 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.849567 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be870e-37e3-4a37-80cd-f4a2642ee303-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.967285 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.967251 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.975281 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.975249 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95wj\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-kube-api-access-m95wj\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.975467 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.975455 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.975581 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.975570 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.984642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.984616 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.984847 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.984831 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:45.988248 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.988224 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be870e-37e3-4a37-80cd-f4a2642ee303-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:46.018041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.893287 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Nov 03 20:52:46.018714 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:45.849258 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.019443 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.019424 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:52:46.047294 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.047246 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.088332 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.088287 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.295638 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.295592 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:46.313326 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.313287 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:46.313630 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.313568 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.345455 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.334132 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:52:46.345998 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.345088 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" event={"ID":"b3be870e-37e3-4a37-80cd-f4a2642ee303","Type":"ContainerDied","Data":"7d0b21f786101bb1f45f4f879a8dc686c9f39644e0ae7febe4b59f94e9e13a1d"} Nov 03 20:52:46.346236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.346215 4227 scope.go:122] "RemoveContainer" containerID="00ce6e13a42bdec6b4fddf7fcab2c277381377591f4e755c898447eb83461b34" Nov 03 20:52:46.346480 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.345185 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b" Nov 03 20:52:46.355321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.345335 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.384619 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.384590 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:52:46.416312 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.416266 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.416636 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.416605 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 03 20:52:46.604455 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.604356 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:46.694578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.641743 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.004184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:46.988820 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.057183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.055426 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 20:52:47.057912 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.057872 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.101247 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.089272 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.303248 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.300309 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.347316 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.347271 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.379186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.375455 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 20:52:47.388111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.388069 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.420544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.420498 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.648246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.645267 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.648246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.647103 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.700303 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.700224 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b"] Nov 03 20:52:47.752212 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.752179 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qtq8b"] Nov 03 20:52:47.977589 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.959861 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:47.977589 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:47.959933 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.120185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.062587 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 20:52:48.120185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.063288 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.120185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.090172 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.120185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.097916 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.166221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.147387 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:48.166221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.147497 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.265239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.247307 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.308183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.304669 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.354791 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.354743 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.387186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.380339 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.437184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.414189 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.455577 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.423996 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.455577 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.424992 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:52:48.455577 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.438056 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:52:48.507328 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.504693 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:52:48.584239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.576926 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:52:48.659046 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.659003 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.782200 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.777121 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:52:48.862997 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:48.862889 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:52:49.029607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.029567 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" path="/var/lib/kubelet/pods/b3be870e-37e3-4a37-80cd-f4a2642ee303/volumes" Nov 03 20:52:49.102301 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.102260 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:49.146260 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.102596 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.146839 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.112243 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.184278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.182319 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:52:49.313293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.313244 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:52:49.343182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.342415 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:49.343182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.342514 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.439193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.428271 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:49.439193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.428335 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.465197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.451959 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.474295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.474255 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:49.474559 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.474525 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.653186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.627974 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 20:52:49.781610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.759448 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.782194 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:49.760968 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.113391 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.113250 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.224185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.223349 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.247277 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.235045 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 20:52:50.314118 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.314076 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.385248 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.384293 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:50.385248 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.384358 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.451626 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.451579 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 20:52:50.473183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.470263 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.473183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.470684 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 20:52:50.741481 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.741450 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:52:50.770535 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.770480 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 20:52:50.771213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.771192 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:52:50.787968 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.787937 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.788659 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.788597 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.793235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.787881 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.793939 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.793918 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:52:50.821315 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.821243 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:52:50.824929 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.824798 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:52:50.885052 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:50.884381 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:52:51.017039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:51.016958 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:52:51.019038 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:51.019015 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:52:53.311774 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:53.311741 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f"] Nov 03 20:52:53.312137 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:53.311925 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" containerID="cri-o://ddf9f18f2d6d96d9aa5c0817353f87d80f605b9bbe49984f3d6112bdd2082e0e" gracePeriod=30 Nov 03 20:52:55.216182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.211230 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj"] Nov 03 20:52:55.216182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.212034 4227 cpu_manager.go:414] "containerMap: removing container" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" Nov 03 20:52:55.216182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.212056 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" Nov 03 20:52:55.217613 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.217575 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3be870e-37e3-4a37-80cd-f4a2642ee303" containerName="istio-proxy" Nov 03 20:52:55.223312 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.223288 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.257678 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.254514 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh"] Nov 03 20:52:55.267182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.259321 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.282225 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.272696 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj"] Nov 03 20:52:55.297933 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.297903 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b119f2-openshift-default-dockercfg-d7fgm\"" Nov 03 20:52:55.362231 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.362191 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.427354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.427319 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.427621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.427606 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tfb\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-kube-api-access-45tfb\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.427722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.427708 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.427807 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.427796 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.427929 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.427918 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f016ad-0799-4093-b879-749bf738eea5-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.428027 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.428015 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.428135 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.428124 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.441448 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.441417 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.441659 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.441641 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.441783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.441767 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.441908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.441892 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.442033 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.442020 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dfn\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-kube-api-access-t2dfn\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.442180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.442150 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.442302 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.442286 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.442407 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.442391 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.442505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.442491 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f016ad-0799-4093-b879-749bf738eea5-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.442605 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.442590 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.464497 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.464398 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh"] Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546432 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dfn\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-kube-api-access-t2dfn\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546503 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546536 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546565 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546592 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f016ad-0799-4093-b879-749bf738eea5-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546615 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546674 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546750 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546825 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tfb\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-kube-api-access-45tfb\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546853 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546882 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546936 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f016ad-0799-4093-b879-749bf738eea5-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.546975 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.547074 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.547178 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.547218 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.547233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.547242 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.548295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.547389 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.559191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.558228 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.559191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.558461 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.559191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.558824 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.561612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.560385 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.561612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.560692 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.561612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.560949 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.561612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.560983 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.561612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.561272 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.561612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.561537 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.564523 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.564477 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.574955 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.564959 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f016ad-0799-4093-b879-749bf738eea5-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.597246 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.597214 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.600244 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.600194 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.602032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.602010 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f016ad-0799-4093-b879-749bf738eea5-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.617144 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.617127 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.617942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.617892 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tfb\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-kube-api-access-45tfb\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.618102 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.618072 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dfn\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-kube-api-access-t2dfn\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.619888 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.619486 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:55.764184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.753514 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:55.880263 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:55.879967 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:56.109601 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:52:56.107781 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:52:56.313774 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:52:56.313738 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefaf6ae3_ef78_49b7_8b0c_b6ee2169102c.slice/crio-bfee8e7fa9934830ef3c79140c62cb52236306da316156f8325dc5eb4e9b8c96 WatchSource:0}: Error finding container bfee8e7fa9934830ef3c79140c62cb52236306da316156f8325dc5eb4e9b8c96: Status 404 returned error can't find the container with id bfee8e7fa9934830ef3c79140c62cb52236306da316156f8325dc5eb4e9b8c96 Nov 03 20:52:56.341525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.317010 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh"] Nov 03 20:52:56.344083 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.344039 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:56.344406 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.344373 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:56.344581 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.344559 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:56.446076 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.445961 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj"] Nov 03 20:52:56.472002 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.471937 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:56.472317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.472029 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:56.472317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.472070 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:52:56.988183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.986038 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" event={"ID":"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c","Type":"ContainerStarted","Data":"47b449a6eeaefbe05f7eae4af8fe2dd81d55e2ff601d0d81dafc698c94e54959"} Nov 03 20:52:56.988183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.986109 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" event={"ID":"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c","Type":"ContainerStarted","Data":"bfee8e7fa9934830ef3c79140c62cb52236306da316156f8325dc5eb4e9b8c96"} Nov 03 20:52:56.992177 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.989176 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" event={"ID":"a5f016ad-0799-4093-b879-749bf738eea5","Type":"ContainerStarted","Data":"3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d"} Nov 03 20:52:56.992177 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:56.989233 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" event={"ID":"a5f016ad-0799-4093-b879-749bf738eea5","Type":"ContainerStarted","Data":"259238a5e0eb9831b71fc61bfeb9aed056d5df058c5d4a75a015c72e6950b972"} Nov 03 20:52:57.103930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:57.103866 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podStartSLOduration=2.1038463099999998 podStartE2EDuration="2.10384631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:57.10047139 +0000 UTC m=+1444.337449268" watchObservedRunningTime="2026-11-03 20:52:57.10384631 +0000 UTC m=+1444.340824188" Nov 03 20:52:57.762877 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:57.762646 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:52:57.779180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:57.776684 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:52:57.779180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:57.776743 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 20:52:57.895452 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:57.895416 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:52:57.934938 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:57.934297 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:52:57.934938 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:57.934361 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:52:58.440641 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:58.440336 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:58.474198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:58.465273 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:58.767188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:58.765945 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:52:58.767188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:58.766003 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 20:52:58.887215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:58.886832 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:52:58.887215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:58.886891 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:52:59.141241 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.138185 4227 generic.go:356] "Generic (PLEG): container finished" podID="9bcd4de8-8914-4263-8300-70c81a055876" containerID="ddf9f18f2d6d96d9aa5c0817353f87d80f605b9bbe49984f3d6112bdd2082e0e" exitCode=0 Nov 03 20:52:59.164196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.161842 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" event={"ID":"9bcd4de8-8914-4263-8300-70c81a055876","Type":"ContainerDied","Data":"ddf9f18f2d6d96d9aa5c0817353f87d80f605b9bbe49984f3d6112bdd2082e0e"} Nov 03 20:52:59.164196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.161915 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" event={"ID":"9bcd4de8-8914-4263-8300-70c81a055876","Type":"ContainerDied","Data":"288f582994b743253667cd62c701026c3fe77bbf4a9255ac8a691d64e403fa62"} Nov 03 20:52:59.164196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.161940 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288f582994b743253667cd62c701026c3fe77bbf4a9255ac8a691d64e403fa62" Nov 03 20:52:59.189415 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.189334 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:52:59.306649 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.306611 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-data\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.306916 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.306892 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-certs\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.307104 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.307087 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-kube-api-access-zvl55\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-kube-api-access-zvl55\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.307243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.307225 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bcd4de8-8914-4263-8300-70c81a055876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd4de8-8914-4263-8300-70c81a055876-istio-podinfo\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.307428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.307411 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bcd4de8-8914-4263-8300-70c81a055876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd4de8-8914-4263-8300-70c81a055876-istiod-ca-cert\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.313184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.307731 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-certs" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.313184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.307873 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcd4de8-8914-4263-8300-70c81a055876-istiod-ca-cert" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:59.313184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.307976 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-data" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.313790 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.313763 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-socket\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.314322 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.314304 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-istio-token\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.315518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.315497 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-envoy\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.315811 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.315793 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-credential-socket\") pod \"9bcd4de8-8914-4263-8300-70c81a055876\" (UID: \"9bcd4de8-8914-4263-8300-70c81a055876\") " Nov 03 20:52:59.316774 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.316749 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.316909 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.316893 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.316998 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.316986 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd4de8-8914-4263-8300-70c81a055876-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.317305 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.314073 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-socket" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.318620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.317750 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bcd4de8-8914-4263-8300-70c81a055876-istio-podinfo" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:59.320126 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.319939 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-envoy" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.320721 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.320685 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-istio-token" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:59.321278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.321240 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-credential-socket" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.322632 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.322614 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-kube-api-access-zvl55" pod "9bcd4de8-8914-4263-8300-70c81a055876" (UID: "9bcd4de8-8914-4263-8300-70c81a055876"). InnerVolumeSpecName "kube-api-access-zvl55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:59.414209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.414134 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podStartSLOduration=4.4141200529999995 podStartE2EDuration="4.414120053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:57.261854837 +0000 UTC m=+1444.498832712" watchObservedRunningTime="2026-11-03 20:52:59.414120053 +0000 UTC m=+1446.651097927" Nov 03 20:52:59.417952 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.417926 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.418112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.418099 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.418203 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.418193 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvl55\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-kube-api-access-zvl55\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.418265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.418249 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd4de8-8914-4263-8300-70c81a055876-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.418327 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.418320 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd4de8-8914-4263-8300-70c81a055876-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.418382 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.418374 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd4de8-8914-4263-8300-70c81a055876-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.767180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.757209 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:52:59.767180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.757271 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 20:52:59.896218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.890090 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:52:59.896218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:52:59.890152 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:53:00.143937 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.143859 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f" Nov 03 20:53:00.326136 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.324556 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f"] Nov 03 20:53:00.331262 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.331233 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jck8f"] Nov 03 20:53:00.758492 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.755080 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:53:00.758492 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.755139 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 20:53:00.886193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.883886 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:53:00.886193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.883969 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:53:00.907006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:00.906977 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcd4de8-8914-4263-8300-70c81a055876" path="/var/lib/kubelet/pods/9bcd4de8-8914-4263-8300-70c81a055876/volumes" Nov 03 20:53:01.760185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:01.757653 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:53:01.760185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:01.757753 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 20:53:01.896181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:01.885333 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:53:01.896181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:01.885393 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:53:02.770369 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:02.770327 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:53:02.770671 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:02.770639 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 20:53:02.884606 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:02.884566 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:53:02.884895 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:02.884865 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:53:03.755112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:03.755069 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:53:03.755506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:03.755147 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 20:53:03.881219 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:03.881150 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:53:03.881486 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:03.881453 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:53:05.529466 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:05.529431 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:53:05.530777 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:05.530749 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:53:05.532148 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:05.532123 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:53:05.727459 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:05.727426 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:53:05.728988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:05.728965 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:53:05.729916 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:05.729896 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:53:22.074129 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.074083 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.092561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.092518 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.168453 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.168390 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.323347 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.323288 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.390731 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.390614 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.422395 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.422335 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.436529 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.435805 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.798322 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.798271 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:22.821183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.798817 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:53:22.847182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.821778 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 20:53:22.971503 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:22.947326 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 20:53:23.078585 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.078463 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.103210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.098740 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.242656 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.241268 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.324411 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.324330 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.366250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.337777 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:53:23.366250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.337844 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:53:23.366250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.337880 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:53:23.366250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.337916 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:53:23.400346 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.399146 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.410276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.410230 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:53:23.566391 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.504767 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.566391 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.510701 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:53:23.566391 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.535279 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.566391 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.539271 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:53:23.847495 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.847448 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz"] Nov 03 20:53:23.900206 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.847808 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:23.900206 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.847852 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:53:23.900206 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.859012 4227 cpu_manager.go:414] "containerMap: removing container" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" Nov 03 20:53:23.900206 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.859039 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" Nov 03 20:53:23.900206 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.859241 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcd4de8-8914-4263-8300-70c81a055876" containerName="istio-proxy" Nov 03 20:53:23.953853 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:23.953817 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:24.121475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.108777 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:24.121475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.109196 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:24.147191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.138384 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:53:24.147191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.138441 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:53:24.147191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.138497 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:53:24.147191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.139385 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.155184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.151294 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 20:53:24.252185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.235106 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00e00e0-ff18-434c-8531-55eccd69c74f-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.258191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.253469 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pxk\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-kube-api-access-c6pxk\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.258729 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.243383 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:24.269188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.259144 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.269188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.259651 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.269188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.259700 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.269188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.260119 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.269188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.260196 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.269188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.260312 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.269188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.260357 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.408486 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.408436 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:24.486185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.485395 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.485574 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.485716 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.485830 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.485946 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00e00e0-ff18-434c-8531-55eccd69c74f-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.486059 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pxk\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-kube-api-access-c6pxk\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486697 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.486203 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486697 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.486316 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.486697 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.486362 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.495362 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.489514 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:24.497576 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.490133 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.497758 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.492953 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.548221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.535240 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.585755 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.559340 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 20:53:24.585755 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.584341 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.585755 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.584855 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.640186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.630237 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00e00e0-ff18-434c-8531-55eccd69c74f-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.649425 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.649398 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.683096 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.659486 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf"] Nov 03 20:53:24.689760 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.687882 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 20:53:24.713502 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.713472 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz"] Nov 03 20:53:24.720764 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.715910 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:53:24.722738 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.722588 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf"] Nov 03 20:53:24.723421 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.722996 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 20:53:24.724846 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.724821 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.725749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.725729 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:53:24.727651 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.726476 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:53:24.754642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.731377 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.754642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.731600 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:53:24.754642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.732655 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:53:24.754642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.750482 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:53:24.754642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.751567 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:53:24.767197 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.766901 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:53:24.775862 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.775808 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pxk\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-kube-api-access-c6pxk\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.800385 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.800361 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.800506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.800423 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.800648 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.800626 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7s7\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-kube-api-access-zk7s7\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.800973 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.800943 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a674663-b61a-44bf-be2e-540c627e852c-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.801120 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.801100 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.801229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.801211 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.801293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.801242 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a674663-b61a-44bf-be2e-540c627e852c-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.801380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.801363 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.801523 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.801473 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.902642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.902607 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.902826 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.902648 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.902826 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.902686 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7s7\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-kube-api-access-zk7s7\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.902826 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.902721 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a674663-b61a-44bf-be2e-540c627e852c-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903022 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.902872 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903022 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.902908 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903022 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.902937 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a674663-b61a-44bf-be2e-540c627e852c-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.903036 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.903064 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.903080 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.903178 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.903253 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903436 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.903399 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.903651 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.903629 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a674663-b61a-44bf-be2e-540c627e852c-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.905131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.905104 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a674663-b61a-44bf-be2e-540c627e852c-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.905258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.905235 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.942179 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.942097 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:24.959045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.959018 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:24.959683 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:24.959665 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7s7\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-kube-api-access-zk7s7\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:25.092621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.092590 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:25.369523 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.369450 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz"] Nov 03 20:53:25.375903 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:53:25.375882 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00e00e0_ff18_434c_8531_55eccd69c74f.slice/crio-308019c84c4a426a80a890b0edff8986a3b5665faa1debfcefdbad3ff9483182 WatchSource:0}: Error finding container 308019c84c4a426a80a890b0edff8986a3b5665faa1debfcefdbad3ff9483182: Status 404 returned error can't find the container with id 308019c84c4a426a80a890b0edff8986a3b5665faa1debfcefdbad3ff9483182 Nov 03 20:53:25.378413 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.378385 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:25.378484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.378443 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:25.378484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.378471 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:25.470870 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.470844 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf"] Nov 03 20:53:25.476376 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:53:25.476351 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a674663_b61a_44bf_be2e_540c627e852c.slice/crio-c2c9c0ed7651abf14454ad5b549e061750e1087ab1eeeed30de3758e394fa6ab WatchSource:0}: Error finding container c2c9c0ed7651abf14454ad5b549e061750e1087ab1eeeed30de3758e394fa6ab: Status 404 returned error can't find the container with id c2c9c0ed7651abf14454ad5b549e061750e1087ab1eeeed30de3758e394fa6ab Nov 03 20:53:25.482590 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.482561 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:25.482677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.482622 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:25.482677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.482653 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:25.789142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.786415 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" event={"ID":"5a674663-b61a-44bf-be2e-540c627e852c","Type":"ContainerStarted","Data":"d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de"} Nov 03 20:53:25.789142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.786519 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" event={"ID":"5a674663-b61a-44bf-be2e-540c627e852c","Type":"ContainerStarted","Data":"c2c9c0ed7651abf14454ad5b549e061750e1087ab1eeeed30de3758e394fa6ab"} Nov 03 20:53:25.790750 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.790714 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" event={"ID":"f00e00e0-ff18-434c-8531-55eccd69c74f","Type":"ContainerStarted","Data":"74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789"} Nov 03 20:53:25.790887 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.790771 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" event={"ID":"f00e00e0-ff18-434c-8531-55eccd69c74f","Type":"ContainerStarted","Data":"308019c84c4a426a80a890b0edff8986a3b5665faa1debfcefdbad3ff9483182"} Nov 03 20:53:25.943042 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:25.943002 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:26.094841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:26.094753 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:26.095505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:26.095177 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:26.095505 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:26.095246 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:26.127883 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:53:26.127843 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:53:26.703219 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:26.687474 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podStartSLOduration=3.687447283 podStartE2EDuration="3.687447283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:26.541467273 +0000 UTC m=+1473.778445148" watchObservedRunningTime="2026-11-03 20:53:26.687447283 +0000 UTC m=+1473.924425159" Nov 03 20:53:26.703219 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:26.702354 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podStartSLOduration=3.702334838 podStartE2EDuration="3.702334838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:26.655493273 +0000 UTC m=+1473.892471148" watchObservedRunningTime="2026-11-03 20:53:26.702334838 +0000 UTC m=+1473.939312718" Nov 03 20:53:26.965230 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:26.962321 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:26.965230 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:26.962398 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:27.206814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.206777 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:27.222492 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.222293 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:27.274188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.272128 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq"] Nov 03 20:53:27.304814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.304784 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.356567 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.356518 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq"] Nov 03 20:53:27.374621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.373567 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1a55d7db-dockercfg-f2bfd\"" Nov 03 20:53:27.374621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.374202 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1a55d7db\"" Nov 03 20:53:27.392105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.382216 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbh2\" (UniqueName: \"kubernetes.io/projected/1e61087c-9b60-4be2-8247-6307be0ef3f0-kube-api-access-xwbh2\") pod \"payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.392105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.382397 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e61087c-9b60-4be2-8247-6307be0ef3f0-plugins-config-volume\") pod \"payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.520467 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.487005 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e61087c-9b60-4be2-8247-6307be0ef3f0-plugins-config-volume\") pod \"payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.524957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.524428 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbh2\" (UniqueName: \"kubernetes.io/projected/1e61087c-9b60-4be2-8247-6307be0ef3f0-kube-api-access-xwbh2\") pod \"payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.525874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.522827 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e61087c-9b60-4be2-8247-6307be0ef3f0-plugins-config-volume\") pod \"payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.599650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.599608 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbh2\" (UniqueName: \"kubernetes.io/projected/1e61087c-9b60-4be2-8247-6307be0ef3f0-kube-api-access-xwbh2\") pod \"payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.800447 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.800352 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:27.944409 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.944374 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:27.944545 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:27.944441 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:28.112356 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:28.112278 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:28.112578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:28.112544 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:28.232551 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:28.232515 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq"] Nov 03 20:53:28.241287 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:53:28.241262 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e61087c_9b60_4be2_8247_6307be0ef3f0.slice/crio-2cdd613ac38a6c241f4c0aa0f02ef839e15a1f50506e2a65f9bd0cf79c4cca9f WatchSource:0}: Error finding container 2cdd613ac38a6c241f4c0aa0f02ef839e15a1f50506e2a65f9bd0cf79c4cca9f: Status 404 returned error can't find the container with id 2cdd613ac38a6c241f4c0aa0f02ef839e15a1f50506e2a65f9bd0cf79c4cca9f Nov 03 20:53:28.242747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:28.242728 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:53:28.852929 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:28.852784 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" event={"ID":"1e61087c-9b60-4be2-8247-6307be0ef3f0","Type":"ContainerStarted","Data":"2cdd613ac38a6c241f4c0aa0f02ef839e15a1f50506e2a65f9bd0cf79c4cca9f"} Nov 03 20:53:28.942586 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:28.942550 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:28.942849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:28.942823 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:29.098398 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:29.097925 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:29.098398 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:29.097985 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:29.996184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:29.955853 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:30.171125 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:30.171085 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:30.171732 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:30.171024 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:30.171988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:30.171949 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:30.942721 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:30.942421 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:30.942721 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:30.942466 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:31.093524 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:31.093482 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:31.094127 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:31.094097 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:31.948661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:31.948619 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:31.948961 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:31.948930 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:32.102216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:32.102142 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:32.102699 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:32.102225 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:32.942873 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:32.942839 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:32.943051 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:32.942895 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:33.093082 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.093039 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:33.093267 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.093096 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:33.203288 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.203210 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9"] Nov 03 20:53:33.213210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.212987 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.270181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.263570 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e20715ef-serving-cert\"" Nov 03 20:53:33.270181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.263916 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9"] Nov 03 20:53:33.327126 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.326495 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hwz\" (UniqueName: \"kubernetes.io/projected/6ef5c613-24f2-4685-aa50-ff2408c4426d-kube-api-access-c2hwz\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.327126 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.326713 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef5c613-24f2-4685-aa50-ff2408c4426d-openshift-service-ca\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.327126 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.326765 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-metrics-certs\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.327126 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.326961 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.428707 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.428662 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.428898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.428803 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hwz\" (UniqueName: \"kubernetes.io/projected/6ef5c613-24f2-4685-aa50-ff2408c4426d-kube-api-access-c2hwz\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.428898 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:53:33.428815 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-serving-cert: secret "maas-api-e2e-migrate-e20715ef-serving-cert" not found Nov 03 20:53:33.428898 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:53:33.428890 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls podName:6ef5c613-24f2-4685-aa50-ff2408c4426d nodeName:}" failed. No retries permitted until 2026-11-03 20:53:33.928868992 +0000 UTC m=+1481.165846846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls") pod "maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" (UID: "6ef5c613-24f2-4685-aa50-ff2408c4426d") : secret "maas-api-e2e-migrate-e20715ef-serving-cert" not found Nov 03 20:53:33.429109 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.428981 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef5c613-24f2-4685-aa50-ff2408c4426d-openshift-service-ca\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.429109 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.429015 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-metrics-certs\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.429964 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.429937 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef5c613-24f2-4685-aa50-ff2408c4426d-openshift-service-ca\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.439864 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.439842 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-metrics-certs\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.504253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.503595 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hwz\" (UniqueName: \"kubernetes.io/projected/6ef5c613-24f2-4685-aa50-ff2408c4426d-kube-api-access-c2hwz\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.942871 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.942835 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 20:53:33.943056 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.942892 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 20:53:33.963349 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.963315 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:33.971876 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:33.971816 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls\") pod \"maas-api-e2e-migrate-e20715ef-7db96865-k4fn9\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:34.094595 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:34.094510 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:53:34.094595 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:34.094565 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 20:53:34.133481 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:34.133446 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:34.592202 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:34.584296 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9"] Nov 03 20:53:34.947642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:34.947606 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:34.948021 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:34.948002 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:34.948767 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:34.948740 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 20:53:35.097717 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:35.097643 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:35.097977 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:35.097931 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:35.098561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:35.098544 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 20:53:35.257862 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:35.257825 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" event={"ID":"1e61087c-9b60-4be2-8247-6307be0ef3f0","Type":"ContainerStarted","Data":"4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a"} Nov 03 20:53:35.258130 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:35.258103 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:35.259593 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:35.259567 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" event={"ID":"6ef5c613-24f2-4685-aa50-ff2408c4426d","Type":"ContainerStarted","Data":"93fd035562db7f9f7240b6f26ab1df2320eaba5cc357f58d4893dc99bbd6c604"} Nov 03 20:53:35.384708 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:35.384312 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" podStartSLOduration=1.837749553 podStartE2EDuration="8.384294914s" totalImagesPullingTime="6.546545361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:35.379755625 +0000 UTC m=+1482.616733500" watchObservedRunningTime="2026-11-03 20:53:35.384294914 +0000 UTC m=+1482.621272789" Nov 03 20:53:36.336722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:36.336683 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" event={"ID":"6ef5c613-24f2-4685-aa50-ff2408c4426d","Type":"ContainerStarted","Data":"5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada"} Nov 03 20:53:36.353219 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:36.339113 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:36.455788 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:36.455731 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" podStartSLOduration=2.565837958 podStartE2EDuration="3.455716219s" totalImagesPullingTime="889.878261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:36.44734259 +0000 UTC m=+1483.684320467" watchObservedRunningTime="2026-11-03 20:53:36.455716219 +0000 UTC m=+1483.692694476" Nov 03 20:53:40.548655 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.548620 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg"] Nov 03 20:53:40.552620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.552598 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.592081 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.592053 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h"] Nov 03 20:53:40.595988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.595973 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.640899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.640863 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.641490 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.641562 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.641632 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.641708 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ec80df-fefb-4a10-9874-922d5447874b-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.641742 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql6d\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-kube-api-access-4ql6d\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.641806 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ec80df-fefb-4a10-9874-922d5447874b-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642487 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.642290 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.642487 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.642371 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.644801 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.644776 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg"] Nov 03 20:53:40.686801 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.686774 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h"] Nov 03 20:53:40.743268 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743240 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.743424 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743281 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743424 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743316 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.743424 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743405 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.743597 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743497 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743597 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743522 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743597 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743558 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql6d\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-kube-api-access-4ql6d\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743597 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743591 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ec80df-fefb-4a10-9874-922d5447874b-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743611 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743641 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743705 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.743803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743744 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.743803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743762 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjft\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-kube-api-access-pbjft\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.744039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743835 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.744039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743909 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.744039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.743970 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.744039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.744023 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ec80df-fefb-4a10-9874-922d5447874b-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.744832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.744063 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.744832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.744133 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.744832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.744212 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.744832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.744242 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.744832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.744671 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.744832 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.744687 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ec80df-fefb-4a10-9874-922d5447874b-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.745975 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.745955 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.747535 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.747517 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ec80df-fefb-4a10-9874-922d5447874b-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.797913 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.797882 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.798045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.798002 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql6d\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-kube-api-access-4ql6d\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:40.845664 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845591 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.845664 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845654 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.845848 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845688 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.845848 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845723 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.845848 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845737 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846008 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845869 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846008 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845928 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjft\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-kube-api-access-pbjft\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846008 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845962 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846008 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.845997 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.846015 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.846026 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846199 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.846084 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846355 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.846324 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.846715 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.846692 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.848308 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.848275 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.848810 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.848789 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.899241 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.899213 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjft\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-kube-api-access-pbjft\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.899354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.899217 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:40.909661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:40.909640 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:41.166237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.166210 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:41.339872 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.339840 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h"] Nov 03 20:53:41.346502 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:53:41.346475 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ec80df_fefb_4a10_9874_922d5447874b.slice/crio-fb12e97baa7996d11b3e1d7e3a1269e2b02d03c4acf28782a2f79ffb2cd16bfd WatchSource:0}: Error finding container fb12e97baa7996d11b3e1d7e3a1269e2b02d03c4acf28782a2f79ffb2cd16bfd: Status 404 returned error can't find the container with id fb12e97baa7996d11b3e1d7e3a1269e2b02d03c4acf28782a2f79ffb2cd16bfd Nov 03 20:53:41.349269 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.349235 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:41.349371 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.349301 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:41.349371 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.349338 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:41.356601 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.356575 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" event={"ID":"d7ec80df-fefb-4a10-9874-922d5447874b","Type":"ContainerStarted","Data":"fb12e97baa7996d11b3e1d7e3a1269e2b02d03c4acf28782a2f79ffb2cd16bfd"} Nov 03 20:53:41.538984 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.538906 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg"] Nov 03 20:53:41.545146 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:53:41.545108 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a533677_a8a8_4a9b_b2ed_01313479f2f0.slice/crio-018a46fe55a72ccfed8462de02b5ef00c3e82761af8af55ad0653110de004f61 WatchSource:0}: Error finding container 018a46fe55a72ccfed8462de02b5ef00c3e82761af8af55ad0653110de004f61: Status 404 returned error can't find the container with id 018a46fe55a72ccfed8462de02b5ef00c3e82761af8af55ad0653110de004f61 Nov 03 20:53:41.547898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.547865 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:41.547956 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.547934 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:41.547995 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:41.547964 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:42.388884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.388846 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" event={"ID":"d7ec80df-fefb-4a10-9874-922d5447874b","Type":"ContainerStarted","Data":"f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a"} Nov 03 20:53:42.393395 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.393362 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" event={"ID":"6a533677-a8a8-4a9b-b2ed-01313479f2f0","Type":"ContainerStarted","Data":"35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb"} Nov 03 20:53:42.393602 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.393584 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" event={"ID":"6a533677-a8a8-4a9b-b2ed-01313479f2f0","Type":"ContainerStarted","Data":"018a46fe55a72ccfed8462de02b5ef00c3e82761af8af55ad0653110de004f61"} Nov 03 20:53:42.521884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.521834 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podStartSLOduration=2.52181992 podStartE2EDuration="2.52181992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:42.517483087 +0000 UTC m=+1489.754460961" watchObservedRunningTime="2026-11-03 20:53:42.52181992 +0000 UTC m=+1489.758797789" Nov 03 20:53:42.639676 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.639579 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podStartSLOduration=2.639562879 podStartE2EDuration="2.639562879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:42.63639545 +0000 UTC m=+1489.873373324" watchObservedRunningTime="2026-11-03 20:53:42.639562879 +0000 UTC m=+1489.876540797" Nov 03 20:53:42.910352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.910327 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:42.911958 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.911936 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:53:42.912063 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:42.911979 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 20:53:43.169233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:43.168365 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:43.178244 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:43.176361 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:53:43.178244 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:43.176475 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 20:53:43.399182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:43.396776 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:53:43.909964 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:43.909926 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:53:43.910137 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:43.909987 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 20:53:44.184409 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:44.184314 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:53:44.184792 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:44.184749 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 20:53:44.910780 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:44.910740 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:53:44.911150 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:44.910798 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 20:53:45.175276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:45.175189 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:53:45.175511 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:45.175483 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 20:53:45.911183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:45.910529 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:53:45.911183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:45.910594 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 20:53:46.174803 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:46.174711 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:53:46.175085 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:46.175037 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 20:53:46.341368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:46.341343 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:53:46.911446 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:46.911357 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:53:46.911446 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:46.911436 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 20:53:47.170699 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:47.170611 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:53:47.170932 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:47.170901 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 20:53:47.910683 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:47.910646 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:53:47.910902 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:47.910710 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 20:53:48.167620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:48.167535 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:53:48.168097 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:48.167610 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 20:53:49.060595 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:49.060542 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:49.061566 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:49.061505 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:49.062914 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:49.062838 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:53:49.172089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:49.172060 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:49.172587 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:49.172379 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:49.173151 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:49.173129 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:53:53.047074 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:53.047042 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:53:53.051109 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:53.051081 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:53:53.071578 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:53.071550 4227 scope.go:122] "RemoveContainer" containerID="ddf9f18f2d6d96d9aa5c0817353f87d80f605b9bbe49984f3d6112bdd2082e0e" Nov 03 20:53:55.660906 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.660868 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk"] Nov 03 20:53:55.668929 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.668907 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.717897 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.717867 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718039 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.717914 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718090 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.718060 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718131 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.718099 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.718128 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f98bb0-8ca4-4687-826c-ed9164a46130-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718365 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.718341 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718414 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.718380 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgcm\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-kube-api-access-trgcm\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.718481 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.718552 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.718536 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.747542 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.747513 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk"] Nov 03 20:53:55.820057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820031 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820063 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820095 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820373 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820237 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820373 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820275 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f98bb0-8ca4-4687-826c-ed9164a46130-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820373 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820359 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820374 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgcm\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-kube-api-access-trgcm\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820512 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820516 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820775 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820596 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820775 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820695 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.820883 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820811 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.821013 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820991 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.821075 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.820999 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f98bb0-8ca4-4687-826c-ed9164a46130-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.822519 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.822498 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.822742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.822727 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.872914 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.872894 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.873148 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.873128 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgcm\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-kube-api-access-trgcm\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:55.983250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:55.983183 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:56.093785 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:53:56.093745 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:53:56.407607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.407573 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk"] Nov 03 20:53:56.415991 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:53:56.415959 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f98bb0_8ca4_4687_826c_ed9164a46130.slice/crio-fd2580f5ee350c7f9671c3de459099651f022c6a9046374b76f2e941a14bc2ec WatchSource:0}: Error finding container fd2580f5ee350c7f9671c3de459099651f022c6a9046374b76f2e941a14bc2ec: Status 404 returned error can't find the container with id fd2580f5ee350c7f9671c3de459099651f022c6a9046374b76f2e941a14bc2ec Nov 03 20:53:56.422841 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.422805 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:56.422953 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.422886 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:56.422953 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.422942 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:53:56.547733 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.547696 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" event={"ID":"44f98bb0-8ca4-4687-826c-ed9164a46130","Type":"ContainerStarted","Data":"5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31"} Nov 03 20:53:56.547887 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.547753 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" event={"ID":"44f98bb0-8ca4-4687-826c-ed9164a46130","Type":"ContainerStarted","Data":"fd2580f5ee350c7f9671c3de459099651f022c6a9046374b76f2e941a14bc2ec"} Nov 03 20:53:56.658375 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.658262 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podStartSLOduration=1.658243739 podStartE2EDuration="1.658243739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:56.657419063 +0000 UTC m=+1503.894396938" watchObservedRunningTime="2026-11-03 20:53:56.658243739 +0000 UTC m=+1503.895221620" Nov 03 20:53:56.984282 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:56.984202 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:53:57.987804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:57.987394 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:53:57.987804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:57.987466 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:53:58.983517 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:58.983483 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:53:58.983716 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:58.983548 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:53:59.983770 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:59.983727 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:53:59.984221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:53:59.983804 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:54:00.989189 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:00.983555 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:54:00.989189 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:00.983621 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:54:01.983888 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:01.983842 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:54:01.984084 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:01.983916 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:54:02.996198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:02.988084 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:54:02.996198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:02.988145 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:54:03.988685 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:03.988651 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:54:03.989079 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:03.989047 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:54:03.989684 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:03.989663 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:54:07.407971 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.407934 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5"] Nov 03 20:54:07.416256 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.415562 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.458063 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.458028 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5"] Nov 03 20:54:07.497122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.497092 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1a55d7db-serving-cert\"" Nov 03 20:54:07.785245 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.759786 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e169736d-5e52-4e6a-beb8-accdd810cca0-openshift-service-ca\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.785570 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.785539 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.785754 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.785735 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqnq\" (UniqueName: \"kubernetes.io/projected/e169736d-5e52-4e6a-beb8-accdd810cca0-kube-api-access-jgqnq\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.827658 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.822213 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-metrics-certs\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.978361 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.978309 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e169736d-5e52-4e6a-beb8-accdd810cca0-openshift-service-ca\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.978739 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.978645 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.979067 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.979017 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqnq\" (UniqueName: \"kubernetes.io/projected/e169736d-5e52-4e6a-beb8-accdd810cca0-kube-api-access-jgqnq\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:07.979436 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:07.979399 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-metrics-certs\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:08.004208 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:54:07.982276 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-serving-cert: secret "maas-api-e2e-derive-1a55d7db-serving-cert" not found Nov 03 20:54:08.004208 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:54:07.982379 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls podName:e169736d-5e52-4e6a-beb8-accdd810cca0 nodeName:}" failed. No retries permitted until 2026-11-03 20:54:08.482353482 +0000 UTC m=+1515.719331337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls") pod "maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" (UID: "e169736d-5e52-4e6a-beb8-accdd810cca0") : secret "maas-api-e2e-derive-1a55d7db-serving-cert" not found Nov 03 20:54:08.006575 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:08.006456 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e169736d-5e52-4e6a-beb8-accdd810cca0-openshift-service-ca\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:08.029185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:08.028110 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-metrics-certs\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:08.098884 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:08.097604 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqnq\" (UniqueName: \"kubernetes.io/projected/e169736d-5e52-4e6a-beb8-accdd810cca0-kube-api-access-jgqnq\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:08.561181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:08.555320 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:08.561181 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:54:08.555651 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-serving-cert: secret "maas-api-e2e-derive-1a55d7db-serving-cert" not found Nov 03 20:54:08.561181 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:54:08.555734 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls podName:e169736d-5e52-4e6a-beb8-accdd810cca0 nodeName:}" failed. No retries permitted until 2026-11-03 20:54:09.555711673 +0000 UTC m=+1516.792689538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls") pod "maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" (UID: "e169736d-5e52-4e6a-beb8-accdd810cca0") : secret "maas-api-e2e-derive-1a55d7db-serving-cert" not found Nov 03 20:54:09.153358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.153281 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 20:54:09.289210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.284866 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 20:54:09.415703 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.415600 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.637024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.636972 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:09.661341 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.661305 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\") pod \"maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:09.730333 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.730238 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 20:54:09.742108 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.742062 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.759357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.759307 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.781216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.780706 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:09.861118 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:09.861086 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:10.147428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.147303 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:10.290933 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.290871 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:10.321198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.291933 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:54:10.598185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.582633 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 20:54:10.860233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.859371 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 20:54:10.899193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.868542 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:54:10.899193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.868606 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:54:10.899193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.887307 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 20:54:10.899193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.887608 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:10.911335 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.900390 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:54:11.469184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.911956 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:54:11.472191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.911995 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.472191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:10.930284 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.492221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:11.148109 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.512185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:11.174235 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:11.542221 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:11.540308 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:54:11.638128 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:54:11.638092 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode169736d_5e52_4e6a_beb8_accdd810cca0.slice/crio-ace8020ac935cd3de8e5d7dd3e31aaaeb4f16572494a0439615754a8bcbbf5ea WatchSource:0}: Error finding container ace8020ac935cd3de8e5d7dd3e31aaaeb4f16572494a0439615754a8bcbbf5ea: Status 404 returned error can't find the container with id ace8020ac935cd3de8e5d7dd3e31aaaeb4f16572494a0439615754a8bcbbf5ea Nov 03 20:54:11.867057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:11.866970 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 03 20:54:11.908082 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:11.898033 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 20:54:11.908082 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:11.898067 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 20:54:12.166518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.166281 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:54:12.166518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.166344 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:54:12.166518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.166376 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 20:54:12.166518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.166407 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 20:54:12.166518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.166446 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:54:12.192260 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.170965 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 20:54:12.197398 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.197370 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 20:54:12.197650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.197633 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 20:54:12.197802 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.197789 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:54:12.197920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.197907 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5"] Nov 03 20:54:12.493516 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:12.493421 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" event={"ID":"e169736d-5e52-4e6a-beb8-accdd810cca0","Type":"ContainerStarted","Data":"ace8020ac935cd3de8e5d7dd3e31aaaeb4f16572494a0439615754a8bcbbf5ea"} Nov 03 20:54:13.501460 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:13.501418 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" event={"ID":"e169736d-5e52-4e6a-beb8-accdd810cca0","Type":"ContainerStarted","Data":"d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a"} Nov 03 20:54:13.501909 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:13.501483 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:13.610597 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:13.610544 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" podStartSLOduration=5.611168458 podStartE2EDuration="6.61053045s" totalImagesPullingTime="999.361992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:13.607644935 +0000 UTC m=+1520.844622848" watchObservedRunningTime="2026-11-03 20:54:13.61053045 +0000 UTC m=+1520.847508324" Nov 03 20:54:19.561093 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:19.561060 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:54:25.895513 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.895479 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8"] Nov 03 20:54:25.900429 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.900397 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.931610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.931572 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.931610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.931608 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871eb20-8eba-4e0a-9271-5e182783dcd3-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.931864 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.931650 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.931864 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.931786 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrjc\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-kube-api-access-vzrjc\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.931864 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.931826 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.932019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.931913 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.932019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.931950 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.932086 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.932048 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:25.932117 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:25.932104 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.011562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.011528 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8"] Nov 03 20:54:26.034362 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.034331 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.034618 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.034597 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871eb20-8eba-4e0a-9271-5e182783dcd3-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.035447 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.035418 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.035544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.035349 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871eb20-8eba-4e0a-9271-5e182783dcd3-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.035544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.034916 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.035544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.035466 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrjc\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-kube-api-access-vzrjc\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.035721 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.035556 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.035882 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.035862 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.036262 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.036235 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.036358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.036265 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.036481 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.036462 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.036561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.036510 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.036623 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.036599 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.036939 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.036915 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.039583 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.039562 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.043804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.043780 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.087188 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:54:26.087142 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:54:26.089765 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.089742 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrjc\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-kube-api-access-vzrjc\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.091826 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.091802 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.214337 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.214307 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:26.648823 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.648737 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8"] Nov 03 20:54:26.654365 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:54:26.654336 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc871eb20_8eba_4e0a_9271_5e182783dcd3.slice/crio-8c0efed8677a2b522200b6a64d58fd4b6bb2e89738da88e6316a567fdc2851ec WatchSource:0}: Error finding container 8c0efed8677a2b522200b6a64d58fd4b6bb2e89738da88e6316a567fdc2851ec: Status 404 returned error can't find the container with id 8c0efed8677a2b522200b6a64d58fd4b6bb2e89738da88e6316a567fdc2851ec Nov 03 20:54:26.657238 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.657206 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:54:26.657304 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.657273 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:54:26.657341 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.657302 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:54:26.696311 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:26.696284 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" event={"ID":"c871eb20-8eba-4e0a-9271-5e182783dcd3","Type":"ContainerStarted","Data":"8c0efed8677a2b522200b6a64d58fd4b6bb2e89738da88e6316a567fdc2851ec"} Nov 03 20:54:27.741243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:27.730400 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" event={"ID":"c871eb20-8eba-4e0a-9271-5e182783dcd3","Type":"ContainerStarted","Data":"8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c"} Nov 03 20:54:28.215098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:28.215068 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:28.216754 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:28.216724 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:54:28.216897 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:28.216800 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:54:29.215007 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:29.214971 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:54:29.215448 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:29.215026 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:54:30.215499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:30.215463 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:54:30.215922 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:30.215524 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:54:31.215580 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:31.215542 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:54:31.216074 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:31.215600 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:54:32.215518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:32.215486 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:54:32.215705 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:32.215544 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:54:33.215004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:33.214968 4227 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:54:33.215193 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:33.215027 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:54:34.219242 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:34.219202 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:34.219726 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:34.219706 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:34.220347 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:34.220327 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:54:34.376168 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:54:34.376112 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podStartSLOduration=9.376096425 podStartE2EDuration="9.376096425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:27.939328842 +0000 UTC m=+1535.176306715" watchObservedRunningTime="2026-11-03 20:54:34.376096425 +0000 UTC m=+1541.613074299" Nov 03 20:54:56.261714 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:54:56.261669 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:55:26.086783 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:55:26.086740 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:55:29.284533 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:29.284493 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq"] Nov 03 20:55:32.461572 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:32.461539 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9"] Nov 03 20:55:32.462181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:32.461826 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" podUID="6ef5c613-24f2-4685-aa50-ff2408c4426d" containerName="maas-api" containerID="cri-o://5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada" gracePeriod=30 Nov 03 20:55:33.170175 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.163449 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:55:33.309213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.309103 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-metrics-certs\") pod \"6ef5c613-24f2-4685-aa50-ff2408c4426d\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " Nov 03 20:55:33.309368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.309229 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef5c613-24f2-4685-aa50-ff2408c4426d-kube-api-access-c2hwz\" (UniqueName: \"kubernetes.io/projected/6ef5c613-24f2-4685-aa50-ff2408c4426d-kube-api-access-c2hwz\") pod \"6ef5c613-24f2-4685-aa50-ff2408c4426d\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " Nov 03 20:55:33.309421 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.309388 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls\") pod \"6ef5c613-24f2-4685-aa50-ff2408c4426d\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " Nov 03 20:55:33.309620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.309599 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef5c613-24f2-4685-aa50-ff2408c4426d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef5c613-24f2-4685-aa50-ff2408c4426d-openshift-service-ca\") pod \"6ef5c613-24f2-4685-aa50-ff2408c4426d\" (UID: \"6ef5c613-24f2-4685-aa50-ff2408c4426d\") " Nov 03 20:55:33.310018 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.309988 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef5c613-24f2-4685-aa50-ff2408c4426d-openshift-service-ca" pod "6ef5c613-24f2-4685-aa50-ff2408c4426d" (UID: "6ef5c613-24f2-4685-aa50-ff2408c4426d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:33.310564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.310539 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef5c613-24f2-4685-aa50-ff2408c4426d-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:33.312333 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.312305 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-metrics-certs" pod "6ef5c613-24f2-4685-aa50-ff2408c4426d" (UID: "6ef5c613-24f2-4685-aa50-ff2408c4426d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:33.312842 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.312816 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef5c613-24f2-4685-aa50-ff2408c4426d-kube-api-access-c2hwz" pod "6ef5c613-24f2-4685-aa50-ff2408c4426d" (UID: "6ef5c613-24f2-4685-aa50-ff2408c4426d"). InnerVolumeSpecName "kube-api-access-c2hwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:33.312946 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.312877 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls" pod "6ef5c613-24f2-4685-aa50-ff2408c4426d" (UID: "6ef5c613-24f2-4685-aa50-ff2408c4426d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:33.338270 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.338241 4227 generic.go:356] "Generic (PLEG): container finished" podID="6ef5c613-24f2-4685-aa50-ff2408c4426d" containerID="5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada" exitCode=0 Nov 03 20:55:33.338404 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.338389 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" event={"ID":"6ef5c613-24f2-4685-aa50-ff2408c4426d","Type":"ContainerDied","Data":"5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada"} Nov 03 20:55:33.338474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.338436 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" event={"ID":"6ef5c613-24f2-4685-aa50-ff2408c4426d","Type":"ContainerDied","Data":"93fd035562db7f9f7240b6f26ab1df2320eaba5cc357f58d4893dc99bbd6c604"} Nov 03 20:55:33.338474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.338463 4227 scope.go:122] "RemoveContainer" containerID="5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada" Nov 03 20:55:33.338595 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.338583 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9" Nov 03 20:55:33.352641 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.352618 4227 scope.go:122] "RemoveContainer" containerID="5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada" Nov 03 20:55:33.352912 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:55:33.352892 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada\": container with ID starting with 5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada not found: ID does not exist" containerID="5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada" Nov 03 20:55:33.352957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.352921 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada"} err="failed to get container status \"5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada\": rpc error: code = NotFound desc = could not find container \"5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada\": container with ID starting with 5ff68c86720758de20abe68f4130519a959e2f8df761b03c397a8797e452bada not found: ID does not exist" Nov 03 20:55:33.411898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.411863 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:33.411898 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.411897 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hwz\" (UniqueName: \"kubernetes.io/projected/6ef5c613-24f2-4685-aa50-ff2408c4426d-kube-api-access-c2hwz\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:33.412109 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.411908 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef5c613-24f2-4685-aa50-ff2408c4426d-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:33.496359 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.496330 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9"] Nov 03 20:55:33.499759 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:33.499730 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e20715ef-7db96865-k4fn9"] Nov 03 20:55:34.285563 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.285514 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" podUID="1e61087c-9b60-4be2-8247-6307be0ef3f0" containerName="payload-processing" containerID="cri-o://4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a" gracePeriod=25 Nov 03 20:55:34.679699 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.679673 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:55:34.723600 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.723573 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e61087c-9b60-4be2-8247-6307be0ef3f0-kube-api-access-xwbh2\" (UniqueName: \"kubernetes.io/projected/1e61087c-9b60-4be2-8247-6307be0ef3f0-kube-api-access-xwbh2\") pod \"1e61087c-9b60-4be2-8247-6307be0ef3f0\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " Nov 03 20:55:34.723747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.723659 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e61087c-9b60-4be2-8247-6307be0ef3f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e61087c-9b60-4be2-8247-6307be0ef3f0-plugins-config-volume\") pod \"1e61087c-9b60-4be2-8247-6307be0ef3f0\" (UID: \"1e61087c-9b60-4be2-8247-6307be0ef3f0\") " Nov 03 20:55:34.724060 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.724038 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e61087c-9b60-4be2-8247-6307be0ef3f0-plugins-config-volume" pod "1e61087c-9b60-4be2-8247-6307be0ef3f0" (UID: "1e61087c-9b60-4be2-8247-6307be0ef3f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:34.724248 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.724232 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e61087c-9b60-4be2-8247-6307be0ef3f0-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:34.725590 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.725565 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e61087c-9b60-4be2-8247-6307be0ef3f0-kube-api-access-xwbh2" pod "1e61087c-9b60-4be2-8247-6307be0ef3f0" (UID: "1e61087c-9b60-4be2-8247-6307be0ef3f0"). InnerVolumeSpecName "kube-api-access-xwbh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:34.830089 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.830052 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwbh2\" (UniqueName: \"kubernetes.io/projected/1e61087c-9b60-4be2-8247-6307be0ef3f0-kube-api-access-xwbh2\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:34.910503 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:34.910469 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef5c613-24f2-4685-aa50-ff2408c4426d" path="/var/lib/kubelet/pods/6ef5c613-24f2-4685-aa50-ff2408c4426d/volumes" Nov 03 20:55:35.352731 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.352648 4227 generic.go:356] "Generic (PLEG): container finished" podID="1e61087c-9b60-4be2-8247-6307be0ef3f0" containerID="4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a" exitCode=0 Nov 03 20:55:35.352731 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.352695 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" Nov 03 20:55:35.352731 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.352719 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" event={"ID":"1e61087c-9b60-4be2-8247-6307be0ef3f0","Type":"ContainerDied","Data":"4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a"} Nov 03 20:55:35.353004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.352761 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq" event={"ID":"1e61087c-9b60-4be2-8247-6307be0ef3f0","Type":"ContainerDied","Data":"2cdd613ac38a6c241f4c0aa0f02ef839e15a1f50506e2a65f9bd0cf79c4cca9f"} Nov 03 20:55:35.353004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.352786 4227 scope.go:122] "RemoveContainer" containerID="4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a" Nov 03 20:55:35.362150 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.362132 4227 scope.go:122] "RemoveContainer" containerID="4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a" Nov 03 20:55:35.362451 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:55:35.362432 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a\": container with ID starting with 4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a not found: ID does not exist" containerID="4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a" Nov 03 20:55:35.362501 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.362460 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a"} err="failed to get container status \"4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a\": rpc error: code = NotFound desc = could not find container \"4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a\": container with ID starting with 4c9b77832de3bfe55f98f73c49321c43ef5f0075a5e9cf46daa4af2bb17bdb7a not found: ID does not exist" Nov 03 20:55:35.507544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.507511 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq"] Nov 03 20:55:35.553208 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:35.553172 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1a55d7db-fb89f7fd5-s5qvq"] Nov 03 20:55:36.910561 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:36.910528 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e61087c-9b60-4be2-8247-6307be0ef3f0" path="/var/lib/kubelet/pods/1e61087c-9b60-4be2-8247-6307be0ef3f0/volumes" Nov 03 20:55:43.851302 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:43.851117 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs"] Nov 03 20:55:43.881196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:43.862101 4227 cpu_manager.go:414] "containerMap: removing container" podUID="1e61087c-9b60-4be2-8247-6307be0ef3f0" containerName="payload-processing" Nov 03 20:55:43.881196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:43.862130 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e61087c-9b60-4be2-8247-6307be0ef3f0" containerName="payload-processing" Nov 03 20:55:43.881196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:43.862148 4227 cpu_manager.go:414] "containerMap: removing container" podUID="6ef5c613-24f2-4685-aa50-ff2408c4426d" containerName="maas-api" Nov 03 20:55:43.881196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:43.862173 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef5c613-24f2-4685-aa50-ff2408c4426d" containerName="maas-api" Nov 03 20:55:43.881196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:43.862368 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e61087c-9b60-4be2-8247-6307be0ef3f0" containerName="payload-processing" Nov 03 20:55:43.881196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:43.862383 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef5c613-24f2-4685-aa50-ff2408c4426d" containerName="maas-api" Nov 03 20:55:44.108094 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.069461 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:44.211391 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.204364 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wv6\" (UniqueName: \"kubernetes.io/projected/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-kube-api-access-s2wv6\") pod \"payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:44.213944 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.213913 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:44.220836 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.220800 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs"] Nov 03 20:55:44.236393 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.236364 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e8374c4\"" Nov 03 20:55:44.236677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.236662 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e8374c4-dockercfg-rkwg9\"" Nov 03 20:55:44.344182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.337559 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:44.344182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.337764 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wv6\" (UniqueName: \"kubernetes.io/projected/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-kube-api-access-s2wv6\") pod \"payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:44.350745 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.350710 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:44.444922 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.444885 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wv6\" (UniqueName: \"kubernetes.io/projected/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-kube-api-access-s2wv6\") pod \"payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:44.482494 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:44.482459 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:45.513183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:45.510984 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs"] Nov 03 20:55:45.594453 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:45.594410 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" event={"ID":"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c","Type":"ContainerStarted","Data":"9b7553fd4c32eaf104f9088d24ca52e182b5810458542784d1aef314cb53305a"} Nov 03 20:55:46.690274 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:46.684068 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" event={"ID":"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c","Type":"ContainerStarted","Data":"0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3"} Nov 03 20:55:46.690274 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:46.685253 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:55:46.799793 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:46.799700 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" podStartSLOduration=3.799686658 podStartE2EDuration="3.799686658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:46.795538281 +0000 UTC m=+1614.032516154" watchObservedRunningTime="2026-11-03 20:55:46.799686658 +0000 UTC m=+1614.036664533" Nov 03 20:55:56.087647 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:55:56.087598 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:55:58.832600 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:55:58.832571 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:56:00.969359 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:00.969317 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5"] Nov 03 20:56:00.969843 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:00.969545 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" podUID="e169736d-5e52-4e6a-beb8-accdd810cca0" containerName="maas-api" containerID="cri-o://d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a" gracePeriod=30 Nov 03 20:56:01.510559 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.498229 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:56:01.636190 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.634833 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e169736d-5e52-4e6a-beb8-accdd810cca0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e169736d-5e52-4e6a-beb8-accdd810cca0-openshift-service-ca\") pod \"e169736d-5e52-4e6a-beb8-accdd810cca0\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " Nov 03 20:56:01.636190 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.634996 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e169736d-5e52-4e6a-beb8-accdd810cca0-kube-api-access-jgqnq\" (UniqueName: \"kubernetes.io/projected/e169736d-5e52-4e6a-beb8-accdd810cca0-kube-api-access-jgqnq\") pod \"e169736d-5e52-4e6a-beb8-accdd810cca0\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " Nov 03 20:56:01.636190 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.635120 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\") pod \"e169736d-5e52-4e6a-beb8-accdd810cca0\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " Nov 03 20:56:01.636190 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.635185 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-metrics-certs\") pod \"e169736d-5e52-4e6a-beb8-accdd810cca0\" (UID: \"e169736d-5e52-4e6a-beb8-accdd810cca0\") " Nov 03 20:56:01.670188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.637976 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e169736d-5e52-4e6a-beb8-accdd810cca0-openshift-service-ca" pod "e169736d-5e52-4e6a-beb8-accdd810cca0" (UID: "e169736d-5e52-4e6a-beb8-accdd810cca0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:01.702470 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.700342 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-metrics-certs" pod "e169736d-5e52-4e6a-beb8-accdd810cca0" (UID: "e169736d-5e52-4e6a-beb8-accdd810cca0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:01.765180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.744845 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e169736d-5e52-4e6a-beb8-accdd810cca0-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:01.765180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.744883 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:01.780179 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.765806 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls" pod "e169736d-5e52-4e6a-beb8-accdd810cca0" (UID: "e169736d-5e52-4e6a-beb8-accdd810cca0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:01.780179 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.765988 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e169736d-5e52-4e6a-beb8-accdd810cca0-kube-api-access-jgqnq" pod "e169736d-5e52-4e6a-beb8-accdd810cca0" (UID: "e169736d-5e52-4e6a-beb8-accdd810cca0"). InnerVolumeSpecName "kube-api-access-jgqnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:01.860448 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.858086 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgqnq\" (UniqueName: \"kubernetes.io/projected/e169736d-5e52-4e6a-beb8-accdd810cca0-kube-api-access-jgqnq\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:01.860448 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.858123 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e169736d-5e52-4e6a-beb8-accdd810cca0-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:01.995899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.987324 4227 generic.go:356] "Generic (PLEG): container finished" podID="e169736d-5e52-4e6a-beb8-accdd810cca0" containerID="d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a" exitCode=0 Nov 03 20:56:01.995899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.987476 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" event={"ID":"e169736d-5e52-4e6a-beb8-accdd810cca0","Type":"ContainerDied","Data":"d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a"} Nov 03 20:56:01.995899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.987533 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" event={"ID":"e169736d-5e52-4e6a-beb8-accdd810cca0","Type":"ContainerDied","Data":"ace8020ac935cd3de8e5d7dd3e31aaaeb4f16572494a0439615754a8bcbbf5ea"} Nov 03 20:56:01.995899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.987566 4227 scope.go:122] "RemoveContainer" containerID="d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a" Nov 03 20:56:01.995899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:01.987729 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5" Nov 03 20:56:02.119782 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:02.119750 4227 scope.go:122] "RemoveContainer" containerID="d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a" Nov 03 20:56:02.120644 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:56:02.120564 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a\": container with ID starting with d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a not found: ID does not exist" containerID="d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a" Nov 03 20:56:02.120644 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:02.120608 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a"} err="failed to get container status \"d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a\": rpc error: code = NotFound desc = could not find container \"d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a\": container with ID starting with d9a3f6bb3a9391dd730fb95929969c3739c63252291a82c8818b7ccb7fd2de1a not found: ID does not exist" Nov 03 20:56:02.292233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:02.282642 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5"] Nov 03 20:56:02.303278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:02.300060 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1a55d7db-687c5dfc9-tvcp5"] Nov 03 20:56:02.902748 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:02.902703 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e169736d-5e52-4e6a-beb8-accdd810cca0" path="/var/lib/kubelet/pods/e169736d-5e52-4e6a-beb8-accdd810cca0/volumes" Nov 03 20:56:09.637569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.637530 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq"] Nov 03 20:56:09.638198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.638176 4227 cpu_manager.go:414] "containerMap: removing container" podUID="e169736d-5e52-4e6a-beb8-accdd810cca0" containerName="maas-api" Nov 03 20:56:09.638250 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.638200 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e169736d-5e52-4e6a-beb8-accdd810cca0" containerName="maas-api" Nov 03 20:56:09.638332 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.638322 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="e169736d-5e52-4e6a-beb8-accdd810cca0" containerName="maas-api" Nov 03 20:56:09.643107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.643088 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.699142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.699114 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq"] Nov 03 20:56:09.738458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738426 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca845828-d40b-4095-93d9-c5f7114bbdff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.738607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738594 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.738748 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738728 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.738831 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738811 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.738891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738871 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.738948 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738919 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfsw\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-kube-api-access-zkfsw\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.739009 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738954 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.739009 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.738992 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.739107 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.739015 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840589 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840558 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca845828-d40b-4095-93d9-c5f7114bbdff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840728 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840669 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840769 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840749 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840771 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840788 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840875 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840809 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfsw\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-kube-api-access-zkfsw\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840875 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840828 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.840875 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840860 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.841020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.840885 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.841073 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.841025 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.841233 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.841138 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.841423 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.841401 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.841510 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.841432 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.841510 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.841487 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca845828-d40b-4095-93d9-c5f7114bbdff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.843125 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.843101 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.843402 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.843384 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.894063 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.893981 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.896279 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.896253 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfsw\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-kube-api-access-zkfsw\") pod \"maas-default-gateway-openshift-default-696f47c6c-7rqsq\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:09.956305 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:09.956285 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:10.386724 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:10.386691 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq"] Nov 03 20:56:10.392972 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:56:10.392944 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca845828_d40b_4095_93d9_c5f7114bbdff.slice/crio-22d455c909a7efd3e3bc4a44dbbae6b3d15c9d554705571319d73aee6fe52170 WatchSource:0}: Error finding container 22d455c909a7efd3e3bc4a44dbbae6b3d15c9d554705571319d73aee6fe52170: Status 404 returned error can't find the container with id 22d455c909a7efd3e3bc4a44dbbae6b3d15c9d554705571319d73aee6fe52170 Nov 03 20:56:10.395527 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:10.395492 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:10.395629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:10.395558 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:10.395629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:10.395587 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:11.077488 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:11.077451 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" event={"ID":"ca845828-d40b-4095-93d9-c5f7114bbdff","Type":"ContainerStarted","Data":"836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3"} Nov 03 20:56:11.077877 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:11.077541 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" event={"ID":"ca845828-d40b-4095-93d9-c5f7114bbdff","Type":"ContainerStarted","Data":"22d455c909a7efd3e3bc4a44dbbae6b3d15c9d554705571319d73aee6fe52170"} Nov 03 20:56:11.238183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:11.238119 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podStartSLOduration=2.238106264 podStartE2EDuration="2.238106264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:11.234430597 +0000 UTC m=+1638.471408471" watchObservedRunningTime="2026-11-03 20:56:11.238106264 +0000 UTC m=+1638.475084137" Nov 03 20:56:11.962659 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:11.962613 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:11.962905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:11.962873 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:56:11.963014 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:11.962936 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:56:12.967181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:12.962251 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:56:12.967181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:12.962315 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:56:13.968184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:13.964575 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:56:13.968184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:13.964640 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:56:14.962313 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:14.962274 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:56:14.962610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:14.962577 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:56:15.959182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:15.957553 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:56:15.959182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:15.957616 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:56:16.956930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:16.956898 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:56:16.957114 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:16.956949 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:56:17.957419 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:17.957383 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:56:17.957880 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:17.957443 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:56:19.223325 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:19.223290 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:19.225506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:19.225481 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:19.233096 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:19.233072 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 20:56:23.507057 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.507022 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf"] Nov 03 20:56:23.511490 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.511470 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.561196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.561155 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6562\" (UniqueName: \"kubernetes.io/projected/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-kube-api-access-x6562\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.561450 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.561430 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.561650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.561609 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-openshift-service-ca\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.561775 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.561652 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-metrics-certs\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.607266 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.607227 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e8374c4-serving-cert\"" Nov 03 20:56:23.614227 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.614199 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf"] Nov 03 20:56:23.665279 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.665234 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6562\" (UniqueName: \"kubernetes.io/projected/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-kube-api-access-x6562\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.666062 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.666033 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.667217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.667194 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-openshift-service-ca\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.668961 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:56:23.666300 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-serving-cert: secret "maas-api-e2e-ait-0e8374c4-serving-cert" not found Nov 03 20:56:23.671342 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.671307 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-openshift-service-ca\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.682473 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.676235 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-metrics-certs\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.682473 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:56:23.676905 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls podName:03b9fce9-2b0e-4cad-ab14-6f707ff2c2be nodeName:}" failed. No retries permitted until 2026-11-03 20:56:24.176881173 +0000 UTC m=+1651.413859027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls") pod "maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" (UID: "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be") : secret "maas-api-e2e-ait-0e8374c4-serving-cert" not found Nov 03 20:56:23.682473 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.682230 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-metrics-certs\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.718316 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.718288 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6562\" (UniqueName: \"kubernetes.io/projected/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-kube-api-access-x6562\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:23.825967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.825883 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx"] Nov 03 20:56:23.830167 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.830135 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.896626 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.896595 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-45f792-openshift-default-dockercfg-qr8p9\"" Nov 03 20:56:23.898769 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.898744 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-envoy\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.898994 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.898978 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-certs\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.899135 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.899121 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4lx\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-kube-api-access-8n4lx\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.899275 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.899261 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb79187f-8c1c-4532-8006-f011d3c72e5e-istiod-ca-cert\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.899380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.899368 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-token\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.899491 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.899477 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-data\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.899603 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.899591 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-socket\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.899702 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.899689 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-podinfo\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.899847 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.899834 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-credential-socket\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:23.905642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:23.905615 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx"] Nov 03 20:56:24.057187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.017742 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4lx\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-kube-api-access-8n4lx\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.069186 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.061983 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb79187f-8c1c-4532-8006-f011d3c72e5e-istiod-ca-cert\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.074305 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.074269 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-token\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.080361 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.080284 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-data\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.080640 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.080611 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-socket\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.080813 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.080798 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-podinfo\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.086181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.080988 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-credential-socket\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.100185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.096463 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-envoy\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.105182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.101314 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-certs\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.105636 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.087462 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-data\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.105779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.087672 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-socket\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.126182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.109578 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-certs\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.126182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.070372 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb79187f-8c1c-4532-8006-f011d3c72e5e-istiod-ca-cert\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.126182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.086839 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-credential-socket\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.126182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.124145 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-podinfo\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.186203 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.184777 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-token\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.186203 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.185533 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4lx\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-kube-api-access-8n4lx\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.186203 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.185730 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-envoy\") pod \"e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.207275 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.207234 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:24.228192 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.228141 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls\") pod \"maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:24.433491 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.433452 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:24.451724 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.451687 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:24.581675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.581616 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq"] Nov 03 20:56:24.647302 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.645782 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l"] Nov 03 20:56:24.647302 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.645985 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.654986 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.654961 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq"] Nov 03 20:56:24.674541 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.655250 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.689068 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.688979 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l"] Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739413 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739468 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739522 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50513e57-b41d-4005-bbfd-956125b1b7dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739556 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739674 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xh5\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-kube-api-access-82xh5\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739716 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739763 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739806 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739833 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739871 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822d773b-ea97-47e6-99d6-9b6b884ea61a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.739927 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.740002 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.740033 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.740061 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.740117 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.740266 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.740303 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdsp\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-kube-api-access-zxdsp\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.742176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.740395 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.841485 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841455 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.841629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841504 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.841629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841537 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.841629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841589 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.841741 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841655 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.841741 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841684 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdsp\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-kube-api-access-zxdsp\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.841804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841752 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.841835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841800 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.841835 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841828 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.841900 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841877 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50513e57-b41d-4005-bbfd-956125b1b7dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.841930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841912 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.841962 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.841936 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.842103 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842086 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xh5\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-kube-api-access-82xh5\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.842150 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842133 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.842215 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842201 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.842262 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842248 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.842322 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842277 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.842322 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842311 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822d773b-ea97-47e6-99d6-9b6b884ea61a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.842399 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842368 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.842651 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842632 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.842836 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842814 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.842953 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.842935 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.843102 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.843080 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.843890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.843863 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50513e57-b41d-4005-bbfd-956125b1b7dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.843994 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.843892 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.844176 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.844130 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.844255 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.844152 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.844817 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.844786 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822d773b-ea97-47e6-99d6-9b6b884ea61a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.846405 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.846380 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.847374 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.847347 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.847883 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.847862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.858389 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.858363 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.900722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.900692 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdsp\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-kube-api-access-zxdsp\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.905959 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.904705 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-9522l\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:24.915146 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.914064 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.932165 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.932130 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xh5\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-kube-api-access-82xh5\") pod \"maas-default-gateway-openshift-default-696f47c6c-98pkq\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:24.981252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:24.981186 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:25.062273 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.062246 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx"] Nov 03 20:56:25.062916 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.062690 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:25.091178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.090879 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.091178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.090963 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.091178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.091007 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.103080 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.103058 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf"] Nov 03 20:56:25.108442 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:56:25.108402 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b9fce9_2b0e_4cad_ab14_6f707ff2c2be.slice/crio-6a0c7536de6585fb1af28ac9fd15ca1d9e7036786dedc3169414beed72538a8b WatchSource:0}: Error finding container 6a0c7536de6585fb1af28ac9fd15ca1d9e7036786dedc3169414beed72538a8b: Status 404 returned error can't find the container with id 6a0c7536de6585fb1af28ac9fd15ca1d9e7036786dedc3169414beed72538a8b Nov 03 20:56:25.268115 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.267957 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" event={"ID":"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be","Type":"ContainerStarted","Data":"6a0c7536de6585fb1af28ac9fd15ca1d9e7036786dedc3169414beed72538a8b"} Nov 03 20:56:25.271196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.270416 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" event={"ID":"cb79187f-8c1c-4532-8006-f011d3c72e5e","Type":"ContainerStarted","Data":"981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102"} Nov 03 20:56:25.271196 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.270464 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" event={"ID":"cb79187f-8c1c-4532-8006-f011d3c72e5e","Type":"ContainerStarted","Data":"6d715b882467b0e2d21afdd579d6eb3cbdbbc111715f538e9849a8669ef94e30"} Nov 03 20:56:25.376695 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.376663 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq"] Nov 03 20:56:25.384446 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.384290 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podStartSLOduration=2.384271408 podStartE2EDuration="2.384271408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:25.380867646 +0000 UTC m=+1652.617845519" watchObservedRunningTime="2026-11-03 20:56:25.384271408 +0000 UTC m=+1652.621249288" Nov 03 20:56:25.386350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.385932 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.386350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.386017 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.386350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.386084 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.456499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.456475 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:25.551204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.547524 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.551204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.547671 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.551204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.547757 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:56:25.564798 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:25.564770 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l"] Nov 03 20:56:26.117070 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:56:26.117017 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:56:26.283354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.282349 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" event={"ID":"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be","Type":"ContainerStarted","Data":"8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2"} Nov 03 20:56:26.283584 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.283560 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:26.285645 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.285611 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" event={"ID":"822d773b-ea97-47e6-99d6-9b6b884ea61a","Type":"ContainerStarted","Data":"674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05"} Nov 03 20:56:26.285787 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.285665 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" event={"ID":"822d773b-ea97-47e6-99d6-9b6b884ea61a","Type":"ContainerStarted","Data":"3b2fd15443f8074831d1212da67c233b5ed4417538b99e951a022b44bd1c479e"} Nov 03 20:56:26.289486 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.289457 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" event={"ID":"50513e57-b41d-4005-bbfd-956125b1b7dd","Type":"ContainerStarted","Data":"b142b6af454f50dc2b3ae279777ffc2b4ffd70e02ec4aef59d3e56c40d547c8f"} Nov 03 20:56:26.289588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.289499 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" event={"ID":"50513e57-b41d-4005-bbfd-956125b1b7dd","Type":"ContainerStarted","Data":"b7501c4bbe3e9870ab0c7b5a61c159178b8dbad9be88fc37d2848fc3751318f0"} Nov 03 20:56:26.396306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.396191 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" podStartSLOduration=2.538637213 podStartE2EDuration="3.396172117s" totalImagesPullingTime="857.534904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:26.389994566 +0000 UTC m=+1653.626972441" watchObservedRunningTime="2026-11-03 20:56:26.396172117 +0000 UTC m=+1653.633149984" Nov 03 20:56:26.461171 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.461123 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:26.498762 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.498704 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podStartSLOduration=2.498688475 podStartE2EDuration="2.498688475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:26.497570224 +0000 UTC m=+1653.734548102" watchObservedRunningTime="2026-11-03 20:56:26.498688475 +0000 UTC m=+1653.735666349" Nov 03 20:56:26.814076 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.813277 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podStartSLOduration=2.813226746 podStartE2EDuration="2.813226746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:26.645301152 +0000 UTC m=+1653.882279027" watchObservedRunningTime="2026-11-03 20:56:26.813226746 +0000 UTC m=+1654.050204621" Nov 03 20:56:26.983801 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.983767 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:26.988892 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.986837 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:26.988892 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:26.986890 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:27.069019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:27.066555 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:27.088184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:27.083308 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:27.088184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:27.083378 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:27.316264 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:27.316227 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:27.317409 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:27.317389 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:56:27.989181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:27.986783 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:27.989181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:27.986841 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:28.070142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:28.068584 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:28.070142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:28.068645 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:28.988970 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:28.987604 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:28.988970 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:28.987665 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:29.074110 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:29.074077 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:29.074392 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:29.074363 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:29.984591 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:29.984553 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:29.984881 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:29.984852 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:30.068183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:30.065625 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:30.068183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:30.065686 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:30.994236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:30.984642 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:30.994236 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:30.984709 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:31.071184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:31.070477 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:31.071184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:31.070534 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:31.987210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:31.985609 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:31.987210 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:31.985674 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:32.066184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:32.065513 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:32.066184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:32.065576 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:32.982632 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:32.982585 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:32.983209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:32.982652 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:33.064093 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:33.064052 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:33.064305 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:33.064121 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:33.327781 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:33.327701 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:56:33.982612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:33.982568 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-98pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:56:33.982823 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:33.982643 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:56:34.064300 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:34.064258 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-9522l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:56:34.064490 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:34.064342 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 20:56:35.644945 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:35.644910 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:35.645437 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:35.645410 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:35.646650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:35.646612 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 20:56:35.716620 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:35.716572 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:35.717064 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:35.717036 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:35.718622 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:35.718597 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 20:56:38.348183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.312213 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr"] Nov 03 20:56:38.457183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.456767 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:38.605266 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.603594 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cc15b1d7-serving-cert\"" Nov 03 20:56:38.636607 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.626966 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:38.786297 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.765863 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:56:38.786297 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.774500 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:38.814436 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.813260 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr"] Nov 03 20:56:38.860428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.859308 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvrz\" (UniqueName: \"kubernetes.io/projected/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-kube-api-access-wsvrz\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:38.860428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.859572 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-openshift-service-ca\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:38.860428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.859659 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-metrics-certs\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:38.860428 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.859818 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:38.956265 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:38.952734 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:39.035970 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.035936 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-metrics-certs\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.036406 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.036381 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.036692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.036658 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvrz\" (UniqueName: \"kubernetes.io/projected/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-kube-api-access-wsvrz\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.052967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.036955 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 20:56:39.053181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.037178 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-openshift-service-ca\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.053414 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:56:39.052764 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-serving-cert: secret "maas-api-e2e-ait-cc15b1d7-serving-cert" not found Nov 03 20:56:39.053830 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:56:39.053780 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls podName:59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d nodeName:}" failed. No retries permitted until 2026-11-03 20:56:39.553755915 +0000 UTC m=+1666.790733783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls") pod "maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" (UID: "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d") : secret "maas-api-e2e-ait-cc15b1d7-serving-cert" not found Nov 03 20:56:39.053830 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.052936 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-openshift-service-ca\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.129998 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.129918 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-metrics-certs\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.168182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.164785 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 03 20:56:39.172180 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.171293 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvrz\" (UniqueName: \"kubernetes.io/projected/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-kube-api-access-wsvrz\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.197179 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.193773 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:39.598828 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.597997 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.633182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.623934 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls\") pod \"maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:39.638761 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.638728 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:39.761826 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:39.760542 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:40.256577 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:40.256489 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr"] Nov 03 20:56:40.261595 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:56:40.261568 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ffc7a6_5c6f_42d7_9e1c_61d7f9e6c71d.slice/crio-9ead712d46234aa364b8c29daebd6ed110f1952cc497bb632eeb38a3f1077b58 WatchSource:0}: Error finding container 9ead712d46234aa364b8c29daebd6ed110f1952cc497bb632eeb38a3f1077b58: Status 404 returned error can't find the container with id 9ead712d46234aa364b8c29daebd6ed110f1952cc497bb632eeb38a3f1077b58 Nov 03 20:56:40.856526 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:40.856478 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" event={"ID":"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d","Type":"ContainerStarted","Data":"9ead712d46234aa364b8c29daebd6ed110f1952cc497bb632eeb38a3f1077b58"} Nov 03 20:56:41.865638 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:41.865598 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" event={"ID":"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d","Type":"ContainerStarted","Data":"7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e"} Nov 03 20:56:41.866037 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:41.865824 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:41.977025 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:41.976955 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" podStartSLOduration=3.054353807 podStartE2EDuration="3.976936006s" totalImagesPullingTime="922.582199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:41.974087207 +0000 UTC m=+1669.211065082" watchObservedRunningTime="2026-11-03 20:56:41.976936006 +0000 UTC m=+1669.213913881" Nov 03 20:56:47.878280 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:56:47.878246 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:56:56.083333 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:56:56.083293 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:57:26.082211 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:57:26.082171 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:57:35.818848 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:35.818814 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs"] Nov 03 20:57:40.819243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:40.819203 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" podUID="52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" containerName="payload-processing" containerID="cri-o://0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3" gracePeriod=25 Nov 03 20:57:41.102140 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.102113 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:57:41.233144 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.233112 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-plugins-config-volume\") pod \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " Nov 03 20:57:41.233337 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.233296 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-kube-api-access-s2wv6\" (UniqueName: \"kubernetes.io/projected/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-kube-api-access-s2wv6\") pod \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\" (UID: \"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c\") " Nov 03 20:57:41.233489 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.233464 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-plugins-config-volume" pod "52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" (UID: "52adf14d-9fa9-4294-b23f-0b9f2c25ba2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:41.234015 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.233991 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.235685 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.235657 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-kube-api-access-s2wv6" pod "52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" (UID: "52adf14d-9fa9-4294-b23f-0b9f2c25ba2c"). InnerVolumeSpecName "kube-api-access-s2wv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:41.335996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.335928 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wv6\" (UniqueName: \"kubernetes.io/projected/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c-kube-api-access-s2wv6\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.625544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.625453 4227 generic.go:356] "Generic (PLEG): container finished" podID="52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" containerID="0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3" exitCode=0 Nov 03 20:57:41.625544 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.625500 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" Nov 03 20:57:41.625782 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.625538 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" event={"ID":"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c","Type":"ContainerDied","Data":"0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3"} Nov 03 20:57:41.625782 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.625591 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs" event={"ID":"52adf14d-9fa9-4294-b23f-0b9f2c25ba2c","Type":"ContainerDied","Data":"9b7553fd4c32eaf104f9088d24ca52e182b5810458542784d1aef314cb53305a"} Nov 03 20:57:41.625782 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.625613 4227 scope.go:122] "RemoveContainer" containerID="0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3" Nov 03 20:57:41.635775 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.635758 4227 scope.go:122] "RemoveContainer" containerID="0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3" Nov 03 20:57:41.635980 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:57:41.635964 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3\": container with ID starting with 0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3 not found: ID does not exist" containerID="0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3" Nov 03 20:57:41.636030 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.635987 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3"} err="failed to get container status \"0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3\": rpc error: code = NotFound desc = could not find container \"0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3\": container with ID starting with 0ec71c73b4365c65334e381382e2e0be2c2956b532eecbb7a72c099effa2e3b3 not found: ID does not exist" Nov 03 20:57:41.779475 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.779443 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs"] Nov 03 20:57:41.815614 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:41.815582 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e8374c4-86c67db555-lcqbs"] Nov 03 20:57:42.910378 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:42.910342 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" path="/var/lib/kubelet/pods/52adf14d-9fa9-4294-b23f-0b9f2c25ba2c/volumes" Nov 03 20:57:53.800030 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:53.799999 4227 generic.go:356] "Generic (PLEG): container finished" podID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerID="981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102" exitCode=0 Nov 03 20:57:53.800622 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:53.800076 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" event={"ID":"cb79187f-8c1c-4532-8006-f011d3c72e5e","Type":"ContainerDied","Data":"981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102"} Nov 03 20:57:53.800622 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:53.800550 4227 scope.go:122] "RemoveContainer" containerID="981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102" Nov 03 20:57:53.801941 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:53.801910 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:57:53.802038 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:53.801976 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:57:53.802038 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:53.802010 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:57:54.808538 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:54.808504 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" event={"ID":"cb79187f-8c1c-4532-8006-f011d3c72e5e","Type":"ContainerStarted","Data":"b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae"} Nov 03 20:57:55.456512 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:55.456478 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:57:55.456886 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:55.456863 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:57:55.457045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:55.457022 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:57:56.125026 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:57:56.124988 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:57:56.466187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:56.460601 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:57:56.466187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:56.460667 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:57:57.321108 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:57.321065 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:57:57.455593 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:57.455559 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:57:57.455869 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:57.455840 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:57:58.493996 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:58.493959 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:57:58.495094 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:58.495055 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:57:59.452588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:59.452551 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:57:59.452868 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:57:59.452837 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:58:00.452779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:00.452742 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:58:00.453447 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:00.453372 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:58:01.458054 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:01.458024 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:58:01.459191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:01.459154 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 20:58:08.493511 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:08.493429 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf"] Nov 03 20:58:08.493891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:08.493637 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" podUID="03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" containerName="maas-api" containerID="cri-o://8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2" gracePeriod=30 Nov 03 20:58:08.975419 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:08.975069 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:58:09.027258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:08.995428 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-openshift-service-ca\") pod \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " Nov 03 20:58:09.027258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:08.995567 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls\") pod \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " Nov 03 20:58:09.027258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:08.995607 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-kube-api-access-x6562\" (UniqueName: \"kubernetes.io/projected/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-kube-api-access-x6562\") pod \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " Nov 03 20:58:09.027258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:08.995819 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-metrics-certs\") pod \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\" (UID: \"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be\") " Nov 03 20:58:09.027258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.013983 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-metrics-certs" pod "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" (UID: "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:09.027258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.020428 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-openshift-service-ca" pod "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" (UID: "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:09.120239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.099983 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:09.120239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.100023 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:09.226278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.222789 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls" pod "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" (UID: "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:09.226278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.224657 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:09.253368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.230638 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-kube-api-access-x6562" pod "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" (UID: "03b9fce9-2b0e-4cad-ab14-6f707ff2c2be"). InnerVolumeSpecName "kube-api-access-x6562". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:09.253368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.248383 4227 generic.go:356] "Generic (PLEG): container finished" podID="03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" containerID="8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2" exitCode=0 Nov 03 20:58:09.253368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.248648 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" event={"ID":"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be","Type":"ContainerDied","Data":"8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2"} Nov 03 20:58:09.253368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.248705 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" event={"ID":"03b9fce9-2b0e-4cad-ab14-6f707ff2c2be","Type":"ContainerDied","Data":"6a0c7536de6585fb1af28ac9fd15ca1d9e7036786dedc3169414beed72538a8b"} Nov 03 20:58:09.253368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.248739 4227 scope.go:122] "RemoveContainer" containerID="8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2" Nov 03 20:58:09.253368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.248929 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf" Nov 03 20:58:09.333290 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.329743 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6562\" (UniqueName: \"kubernetes.io/projected/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be-kube-api-access-x6562\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:09.353846 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.352321 4227 scope.go:122] "RemoveContainer" containerID="8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2" Nov 03 20:58:09.356603 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:09.354947 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2\": container with ID starting with 8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2 not found: ID does not exist" containerID="8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2" Nov 03 20:58:09.356603 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.354991 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2"} err="failed to get container status \"8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2\": rpc error: code = NotFound desc = could not find container \"8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2\": container with ID starting with 8a02f697c93746601e22ce1d8265dfd8b1d55b7551784886679236a7479908e2 not found: ID does not exist" Nov 03 20:58:09.494299 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.494259 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf"] Nov 03 20:58:09.525228 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:09.511744 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e8374c4-789cbdbbc9-fgnkf"] Nov 03 20:58:10.904178 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:10.904129 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" path="/var/lib/kubelet/pods/03b9fce9-2b0e-4cad-ab14-6f707ff2c2be/volumes" Nov 03 20:58:14.022837 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.022803 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2"] Nov 03 20:58:14.023732 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.023711 4227 cpu_manager.go:414] "containerMap: removing container" podUID="03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" containerName="maas-api" Nov 03 20:58:14.023779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.023736 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" containerName="maas-api" Nov 03 20:58:14.023779 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.023775 4227 cpu_manager.go:414] "containerMap: removing container" podUID="52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" containerName="payload-processing" Nov 03 20:58:14.023840 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.023784 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" containerName="payload-processing" Nov 03 20:58:14.023942 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.023929 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b9fce9-2b0e-4cad-ab14-6f707ff2c2be" containerName="maas-api" Nov 03 20:58:14.023976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.023951 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="52adf14d-9fa9-4294-b23f-0b9f2c25ba2c" containerName="payload-processing" Nov 03 20:58:14.028310 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.028289 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.085091 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.085013 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2"] Nov 03 20:58:14.123604 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.123554 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-b6d800-openshift-default-dockercfg-cxrtv\"" Nov 03 20:58:14.141621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.141589 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-credential-socket\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.141847 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.141822 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-data\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.141976 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.141892 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-socket\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.142036 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.142001 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-token\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.142096 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.142050 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-certs\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.142212 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.142155 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-podinfo\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.142322 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.142254 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbea0aa-e253-4668-bdb9-faa486867e9c-istiod-ca-cert\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.142386 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.142327 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtz9b\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-kube-api-access-gtz9b\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.142386 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.142372 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-envoy\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248018 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-certs\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248171 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-podinfo\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248220 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbea0aa-e253-4668-bdb9-faa486867e9c-istiod-ca-cert\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248322 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtz9b\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-kube-api-access-gtz9b\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248417 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-envoy\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248577 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-credential-socket\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248611 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-data\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248672 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-socket\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.248735 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-token\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.249633 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-certs\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.252363 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-socket\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253184 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.252782 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-credential-socket\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.253981 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.253496 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbea0aa-e253-4668-bdb9-faa486867e9c-istiod-ca-cert\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.259337 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.259307 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-data\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.292235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.291248 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-podinfo\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.322182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.320463 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-envoy\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.334190 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.326941 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtz9b\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-kube-api-access-gtz9b\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.341648 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.339656 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-token\") pod \"e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.352288 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.351948 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:14.882092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.882057 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2"] Nov 03 20:58:14.886177 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:58:14.886139 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbea0aa_e253_4668_bdb9_faa486867e9c.slice/crio-9230ba9a8e899989f66c0d86a46020dd303885890fde0eb35188dc5b3949ab09 WatchSource:0}: Error finding container 9230ba9a8e899989f66c0d86a46020dd303885890fde0eb35188dc5b3949ab09: Status 404 returned error can't find the container with id 9230ba9a8e899989f66c0d86a46020dd303885890fde0eb35188dc5b3949ab09 Nov 03 20:58:14.889361 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.889322 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:14.889472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.889409 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:14.889472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:14.889459 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:15.371295 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:15.371259 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" event={"ID":"ddbea0aa-e253-4668-bdb9-faa486867e9c","Type":"ContainerStarted","Data":"15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548"} Nov 03 20:58:15.371780 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:15.371319 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" event={"ID":"ddbea0aa-e253-4668-bdb9-faa486867e9c","Type":"ContainerStarted","Data":"9230ba9a8e899989f66c0d86a46020dd303885890fde0eb35188dc5b3949ab09"} Nov 03 20:58:15.482086 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:15.482027 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podStartSLOduration=2.482013678 podStartE2EDuration="2.482013678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:15.47971756 +0000 UTC m=+1762.716695433" watchObservedRunningTime="2026-11-03 20:58:15.482013678 +0000 UTC m=+1762.718991550" Nov 03 20:58:16.355493 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:16.355453 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:16.361502 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:16.361472 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:16.378194 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:16.378145 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:16.378926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:16.378904 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 20:58:25.464182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:25.464032 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr"] Nov 03 20:58:25.493183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:25.471543 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" podUID="59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" containerName="maas-api" containerID="cri-o://7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e" gracePeriod=30 Nov 03 20:58:26.132491 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:26.132390 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:58:26.689400 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.689030 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:58:26.690780 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.690748 4227 generic.go:356] "Generic (PLEG): container finished" podID="59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" containerID="7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e" exitCode=0 Nov 03 20:58:26.691141 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.691119 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" event={"ID":"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d","Type":"ContainerDied","Data":"7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e"} Nov 03 20:58:26.691354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.691337 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" event={"ID":"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d","Type":"ContainerDied","Data":"9ead712d46234aa364b8c29daebd6ed110f1952cc497bb632eeb38a3f1077b58"} Nov 03 20:58:26.691458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.691444 4227 scope.go:122] "RemoveContainer" containerID="7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e" Nov 03 20:58:26.739410 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.738938 4227 scope.go:122] "RemoveContainer" containerID="7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e" Nov 03 20:58:26.739776 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:26.739681 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e\": container with ID starting with 7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e not found: ID does not exist" containerID="7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e" Nov 03 20:58:26.739776 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.739721 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e"} err="failed to get container status \"7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e\": rpc error: code = NotFound desc = could not find container \"7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e\": container with ID starting with 7a2362fa1f8782219114747e55be57887a7c147e9a47252607db7d87b4ad362e not found: ID does not exist" Nov 03 20:58:26.841661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.840498 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-metrics-certs\") pod \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " Nov 03 20:58:26.841661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.840555 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-kube-api-access-wsvrz\" (UniqueName: \"kubernetes.io/projected/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-kube-api-access-wsvrz\") pod \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " Nov 03 20:58:26.841661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.840650 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-openshift-service-ca\") pod \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " Nov 03 20:58:26.841661 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.840868 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls\") pod \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\" (UID: \"59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d\") " Nov 03 20:58:26.842592 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.842543 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-openshift-service-ca" pod "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" (UID: "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:26.886962 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.876985 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls" pod "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" (UID: "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:26.886962 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.877123 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-kube-api-access-wsvrz" pod "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" (UID: "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d"). InnerVolumeSpecName "kube-api-access-wsvrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:26.986465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.955330 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:26.986465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.955365 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvrz\" (UniqueName: \"kubernetes.io/projected/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-kube-api-access-wsvrz\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:26.986465 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.955382 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:26.988842 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:26.987316 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-metrics-certs" pod "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" (UID: "59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:27.057412 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:27.057373 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:27.744181 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:27.738208 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr" Nov 03 20:58:27.761179 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:27.753788 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7"] Nov 03 20:58:27.761179 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:27.754104 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" containerID="cri-o://f46b9b33f6dbe12f1db92082d86c409a1f87ea07835731e0e9f6806747b4c417" gracePeriod=30 Nov 03 20:58:28.042538 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:28.042500 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr"] Nov 03 20:58:28.045931 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:28.045905 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc15b1d7-784878fd89-snpcr"] Nov 03 20:58:28.903177 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:28.903123 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" path="/var/lib/kubelet/pods/59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d/volumes" Nov 03 20:58:32.851119 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:32.851075 4227 generic.go:356] "Generic (PLEG): container finished" podID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerID="f46b9b33f6dbe12f1db92082d86c409a1f87ea07835731e0e9f6806747b4c417" exitCode=0 Nov 03 20:58:32.851553 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:32.851148 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" event={"ID":"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed","Type":"ContainerDied","Data":"f46b9b33f6dbe12f1db92082d86c409a1f87ea07835731e0e9f6806747b4c417"} Nov 03 20:58:33.246104 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.246071 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:58:33.246318 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.246129 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 20:58:33.640209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.640184 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:58:33.705027 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.704992 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-credential-socket\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705030 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-token\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-token\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705123 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-data\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705188 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-socket\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705418 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705254 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-envoy\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705418 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705354 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-kube-api-access-snbn7\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-kube-api-access-snbn7\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705418 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705365 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-credential-socket" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.705587 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705421 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-data" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.705587 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705546 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-certs\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705690 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705580 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-socket" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.705690 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705626 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-podinfo\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705690 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705668 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istiod-ca-cert\") pod \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\" (UID: \"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed\") " Nov 03 20:58:33.705837 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.705741 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-certs" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.706114 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.706084 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istiod-ca-cert" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:33.706479 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.706456 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.706564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.706483 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.706564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.706498 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.706564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.706511 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.706564 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.706525 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.707657 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.707632 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-token" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:33.707752 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.707687 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-kube-api-access-snbn7" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "kube-api-access-snbn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:33.707817 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.707799 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-envoy" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:33.708483 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.708464 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-podinfo" pod "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" (UID: "0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:33.809293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.809213 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.809293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.809247 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.809293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.809264 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snbn7\" (UniqueName: \"kubernetes.io/projected/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-kube-api-access-snbn7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.809293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.809279 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:33.862897 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.862868 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" Nov 03 20:58:33.863411 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.862898 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7" event={"ID":"0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed","Type":"ContainerDied","Data":"24b3621c5ff5ad1a22fb3813603e45443ee51e5eb9403afc73ee9438100b0f55"} Nov 03 20:58:33.863626 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:33.863607 4227 scope.go:122] "RemoveContainer" containerID="f46b9b33f6dbe12f1db92082d86c409a1f87ea07835731e0e9f6806747b4c417" Nov 03 20:58:34.022105 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:34.022070 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7"] Nov 03 20:58:34.025453 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:34.025424 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-cqsm7"] Nov 03 20:58:34.903419 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:34.903386 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" path="/var/lib/kubelet/pods/0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed/volumes" Nov 03 20:58:42.822183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:42.817598 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h"] Nov 03 20:58:42.822183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:42.817863 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" containerID="cri-o://f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a" gracePeriod=30 Nov 03 20:58:42.883182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:42.882206 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh"] Nov 03 20:58:42.883182 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:42.882467 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" containerID="cri-o://47b449a6eeaefbe05f7eae4af8fe2dd81d55e2ff601d0d81dafc698c94e54959" gracePeriod=30 Nov 03 20:58:42.905820 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:42.905435 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z"] Nov 03 20:58:42.919952 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:42.906230 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" containerID="cri-o://458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc" gracePeriod=30 Nov 03 20:58:43.195747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:43.194694 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5"] Nov 03 20:58:43.195747 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:43.194948 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" containerID="cri-o://9491ab689dfd4f9de1ac69f8d856f18fbb921601b3f6e2d73e46f35a2aaf222d" gracePeriod=30 Nov 03 20:58:45.076434 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.076370 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6"] Nov 03 20:58:45.076915 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.076627 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" containerID="cri-o://ddf2638fd38ede1682a6359028be9dad296db3dda8c7a7246735883b8d515728" gracePeriod=30 Nov 03 20:58:45.136923 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.136888 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb"] Nov 03 20:58:45.137149 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.137122 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" containerID="cri-o://1a33dd01f50d16647487b9cbb673725c2304021151edf078c42242042d13a93b" gracePeriod=30 Nov 03 20:58:45.138939 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.138906 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr"] Nov 03 20:58:45.139134 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.139110 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" containerID="cri-o://4380218ce1864ba5577416313d5480933dc045baf3c63202f93cba2ee03246cc" gracePeriod=30 Nov 03 20:58:45.182134 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.182090 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg"] Nov 03 20:58:45.182352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.182170 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm"] Nov 03 20:58:45.182352 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.182334 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" containerID="cri-o://3deb5f5a2782f1a988cf54e2b22e6db4f91f00defd347216bfe0f2721d596683" gracePeriod=30 Nov 03 20:58:45.182535 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.182516 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" containerID="cri-o://5362962e9f11fb11fb9e606c01e7a8d73c7f70385ce01e025e8cf379992e2dd6" gracePeriod=30 Nov 03 20:58:45.227078 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.227044 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:45.227240 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:45.227114 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:48.087808 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.087460 4227 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 20:58:48.087808 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.087523 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 20:58:48.103966 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.103938 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:58:48.210416 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210384 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-data\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.210602 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210500 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ec80df-fefb-4a10-9874-922d5447874b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ec80df-fefb-4a10-9874-922d5447874b-istiod-ca-cert\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.210602 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210534 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-envoy\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.210602 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210560 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-istio-token\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-istio-token\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.210757 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210655 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-certs\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.210757 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210667 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-data" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.210868 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210772 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-credential-socket\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.210926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210872 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7ec80df-fefb-4a10-9874-922d5447874b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ec80df-fefb-4a10-9874-922d5447874b-istio-podinfo\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.210926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210900 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-socket\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.211032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210929 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-kube-api-access-4ql6d\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-kube-api-access-4ql6d\") pod \"d7ec80df-fefb-4a10-9874-922d5447874b\" (UID: \"d7ec80df-fefb-4a10-9874-922d5447874b\") " Nov 03 20:58:48.211032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.210993 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-certs" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.211032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.211008 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ec80df-fefb-4a10-9874-922d5447874b-istiod-ca-cert" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.211258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.211138 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-credential-socket" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.211258 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.211248 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-socket" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.211814 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.211787 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.211921 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.211816 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.212030 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.211830 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.212030 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.212026 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.212204 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.212042 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ec80df-fefb-4a10-9874-922d5447874b-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.213479 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.213449 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-envoy" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.213591 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.213563 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7ec80df-fefb-4a10-9874-922d5447874b-istio-podinfo" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.213673 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.213650 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-istio-token" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.213753 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.213733 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-kube-api-access-4ql6d" pod "d7ec80df-fefb-4a10-9874-922d5447874b" (UID: "d7ec80df-fefb-4a10-9874-922d5447874b"). InnerVolumeSpecName "kube-api-access-4ql6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.271029 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.270823 4227 generic.go:356] "Generic (PLEG): container finished" podID="d7ec80df-fefb-4a10-9874-922d5447874b" containerID="f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a" exitCode=0 Nov 03 20:58:48.271029 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.271016 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" event={"ID":"d7ec80df-fefb-4a10-9874-922d5447874b","Type":"ContainerDied","Data":"f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a"} Nov 03 20:58:48.271307 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.271081 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" event={"ID":"d7ec80df-fefb-4a10-9874-922d5447874b","Type":"ContainerDied","Data":"fb12e97baa7996d11b3e1d7e3a1269e2b02d03c4acf28782a2f79ffb2cd16bfd"} Nov 03 20:58:48.271307 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.271129 4227 scope.go:122] "RemoveContainer" containerID="f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a" Nov 03 20:58:48.271307 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.271029 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h" Nov 03 20:58:48.279609 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.279127 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:58:48.281447 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.281416 4227 generic.go:356] "Generic (PLEG): container finished" podID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerID="47b449a6eeaefbe05f7eae4af8fe2dd81d55e2ff601d0d81dafc698c94e54959" exitCode=0 Nov 03 20:58:48.281571 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.281521 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" event={"ID":"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c","Type":"ContainerDied","Data":"47b449a6eeaefbe05f7eae4af8fe2dd81d55e2ff601d0d81dafc698c94e54959"} Nov 03 20:58:48.283376 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.283353 4227 generic.go:356] "Generic (PLEG): container finished" podID="4b690173-19d4-4bdc-8606-3a144223e616" containerID="9491ab689dfd4f9de1ac69f8d856f18fbb921601b3f6e2d73e46f35a2aaf222d" exitCode=0 Nov 03 20:58:48.283876 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.283833 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" event={"ID":"4b690173-19d4-4bdc-8606-3a144223e616","Type":"ContainerDied","Data":"9491ab689dfd4f9de1ac69f8d856f18fbb921601b3f6e2d73e46f35a2aaf222d"} Nov 03 20:58:48.286891 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.286863 4227 generic.go:356] "Generic (PLEG): container finished" podID="b472214b-3588-487d-be6e-2004270492cb" containerID="458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc" exitCode=0 Nov 03 20:58:48.286991 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.286956 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" event={"ID":"b472214b-3588-487d-be6e-2004270492cb","Type":"ContainerDied","Data":"458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc"} Nov 03 20:58:48.287051 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.286993 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" event={"ID":"b472214b-3588-487d-be6e-2004270492cb","Type":"ContainerDied","Data":"bea9acd60c785c378cc6777309808bae07230979fe69cc4fea2a6a986f3bcac8"} Nov 03 20:58:48.287111 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.287057 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z" Nov 03 20:58:48.295457 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.293264 4227 scope.go:122] "RemoveContainer" containerID="f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a" Nov 03 20:58:48.295457 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:48.293593 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a\": container with ID starting with f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a not found: ID does not exist" containerID="f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a" Nov 03 20:58:48.295457 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.293621 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a"} err="failed to get container status \"f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a\": rpc error: code = NotFound desc = could not find container \"f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a\": container with ID starting with f6c18b47734c64b778f4fceb2619a3ac0dfd52c87ad8204fd1dd13f5271b982a not found: ID does not exist" Nov 03 20:58:48.295457 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.293642 4227 scope.go:122] "RemoveContainer" containerID="458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc" Nov 03 20:58:48.308137 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.308016 4227 scope.go:122] "RemoveContainer" containerID="458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc" Nov 03 20:58:48.308612 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:48.308581 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc\": container with ID starting with 458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc not found: ID does not exist" containerID="458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc" Nov 03 20:58:48.308727 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.308621 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc"} err="failed to get container status \"458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc\": rpc error: code = NotFound desc = could not find container \"458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc\": container with ID starting with 458c2aebe3205935e41cf0b2e0214baed1ef85899532ffc76d84a2dbad7791cc not found: ID does not exist" Nov 03 20:58:48.313408 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.313385 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ec80df-fefb-4a10-9874-922d5447874b-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.313562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.313411 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ql6d\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-kube-api-access-4ql6d\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.313562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.313428 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ec80df-fefb-4a10-9874-922d5447874b-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.313562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.313443 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ec80df-fefb-4a10-9874-922d5447874b-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.414818 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.414777 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-istio-token\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-istio-token\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.414891 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-envoy\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.414942 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b472214b-3588-487d-be6e-2004270492cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b472214b-3588-487d-be6e-2004270492cb-istio-podinfo\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415121 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.415038 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-kube-api-access-qj7nk\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-kube-api-access-qj7nk\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.415117 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-certs\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.415190 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-credential-socket\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415317 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.415256 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-data\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415370 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.415313 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b472214b-3588-487d-be6e-2004270492cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b472214b-3588-487d-be6e-2004270492cb-istiod-ca-cert\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.415370 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.415346 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-socket\") pod \"b472214b-3588-487d-be6e-2004270492cb\" (UID: \"b472214b-3588-487d-be6e-2004270492cb\") " Nov 03 20:58:48.416231 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.416202 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-socket" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.417324 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.417254 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-data" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.417562 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.417490 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-credential-socket" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.417650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.417596 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-certs" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.417650 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.417490 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b472214b-3588-487d-be6e-2004270492cb-istiod-ca-cert" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.418054 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.418030 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-istio-token" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.420585 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.420555 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-envoy" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.420691 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.420654 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b472214b-3588-487d-be6e-2004270492cb-istio-podinfo" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.422785 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.422765 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-kube-api-access-qj7nk" pod "b472214b-3588-487d-be6e-2004270492cb" (UID: "b472214b-3588-487d-be6e-2004270492cb"). InnerVolumeSpecName "kube-api-access-qj7nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.454030 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.454001 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:58:48.459037 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.459017 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:58:48.516819 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.516786 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-data\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.516865 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-kube-api-access-45tfb\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-kube-api-access-45tfb\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.516904 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-envoy\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.516942 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-envoy\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.516999 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-kube-api-access-k29dq\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-kube-api-access-k29dq\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517284 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517041 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-istio-token\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-istio-token\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517284 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517096 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b690173-19d4-4bdc-8606-3a144223e616-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b690173-19d4-4bdc-8606-3a144223e616-istio-podinfo\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517284 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517190 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-data" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.517284 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517253 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-socket\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517333 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-credential-socket\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517389 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-certs\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517484 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517438 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istiod-ca-cert\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517635 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517513 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-data\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517635 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517551 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b690173-19d4-4bdc-8606-3a144223e616-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b690173-19d4-4bdc-8606-3a144223e616-istiod-ca-cert\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.517635 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517594 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-token\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-token\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517772 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517643 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-certs\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517780 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-credential-socket" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.517874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517851 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-socket\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.517926 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517904 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-credential-socket\") pod \"4b690173-19d4-4bdc-8606-3a144223e616\" (UID: \"4b690173-19d4-4bdc-8606-3a144223e616\") " Nov 03 20:58:48.518007 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517992 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-podinfo\") pod \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\" (UID: \"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c\") " Nov 03 20:58:48.518067 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517901 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-socket" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.518067 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.517992 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-data" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.518067 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518009 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-certs" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.518283 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518263 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-socket" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.518392 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518367 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istiod-ca-cert" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.518449 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518391 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-certs" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.518449 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518371 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-credential-socket" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.518608 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518454 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b690173-19d4-4bdc-8606-3a144223e616-istiod-ca-cert" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:48.518830 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518806 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.518908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518835 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.518908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518853 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.518908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518869 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.518908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518884 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.518908 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518898 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b472214b-3588-487d-be6e-2004270492cb-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518912 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518926 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518941 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518957 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518971 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b472214b-3588-487d-be6e-2004270492cb-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.518986 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.519001 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b472214b-3588-487d-be6e-2004270492cb-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.519015 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.519029 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.519043 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.519057 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b690173-19d4-4bdc-8606-3a144223e616-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.519074 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj7nk\" (UniqueName: \"kubernetes.io/projected/b472214b-3588-487d-be6e-2004270492cb-kube-api-access-qj7nk\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.519209 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.519090 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.520137 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.520095 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-kube-api-access-45tfb" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "kube-api-access-45tfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.520446 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.520408 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b690173-19d4-4bdc-8606-3a144223e616-istio-podinfo" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.520559 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.520482 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-kube-api-access-k29dq" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "kube-api-access-k29dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.520673 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.520651 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-istio-token" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.520734 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.520674 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-token" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:48.521032 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.521008 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-podinfo" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:48.521274 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.521250 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-envoy" pod "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" (UID: "efaf6ae3-ef78-49b7-8b0c-b6ee2169102c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.521783 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.521764 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-envoy" pod "4b690173-19d4-4bdc-8606-3a144223e616" (UID: "4b690173-19d4-4bdc-8606-3a144223e616"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:48.528093 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.528072 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:48.528212 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.528128 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:48.538172 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.538139 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h"] Nov 03 20:58:48.542693 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.542668 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cgs2h"] Nov 03 20:58:48.621024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.620989 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45tfb\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-kube-api-access-45tfb\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.621024 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.621022 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b690173-19d4-4bdc-8606-3a144223e616-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.621276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.621033 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.621276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.621042 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k29dq\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-kube-api-access-k29dq\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.621276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.621051 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b690173-19d4-4bdc-8606-3a144223e616-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.621276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.621061 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b690173-19d4-4bdc-8606-3a144223e616-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.621276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.621071 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.621276 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.621080 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:48.871559 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.871457 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z"] Nov 03 20:58:48.876613 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.876579 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-7fh7z"] Nov 03 20:58:48.916312 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.916262 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b472214b-3588-487d-be6e-2004270492cb" path="/var/lib/kubelet/pods/b472214b-3588-487d-be6e-2004270492cb/volumes" Nov 03 20:58:48.917218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:48.917179 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" path="/var/lib/kubelet/pods/d7ec80df-fefb-4a10-9874-922d5447874b/volumes" Nov 03 20:58:49.295643 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.295613 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" Nov 03 20:58:49.296132 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.295613 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh" event={"ID":"efaf6ae3-ef78-49b7-8b0c-b6ee2169102c","Type":"ContainerDied","Data":"bfee8e7fa9934830ef3c79140c62cb52236306da316156f8325dc5eb4e9b8c96"} Nov 03 20:58:49.296132 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.295774 4227 scope.go:122] "RemoveContainer" containerID="47b449a6eeaefbe05f7eae4af8fe2dd81d55e2ff601d0d81dafc698c94e54959" Nov 03 20:58:49.297628 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.297243 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" event={"ID":"4b690173-19d4-4bdc-8606-3a144223e616","Type":"ContainerDied","Data":"6542b19c632699a63f664251338938b34726a9bb0ec76bf4c4431316d6aa0c13"} Nov 03 20:58:49.297628 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.297340 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5" Nov 03 20:58:49.307098 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.307076 4227 scope.go:122] "RemoveContainer" containerID="9491ab689dfd4f9de1ac69f8d856f18fbb921601b3f6e2d73e46f35a2aaf222d" Nov 03 20:58:49.451763 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.451728 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5"] Nov 03 20:58:49.457151 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.457124 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sffg5"] Nov 03 20:58:49.557844 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.557736 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh"] Nov 03 20:58:49.561065 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:49.561039 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-sqnsh"] Nov 03 20:58:50.284530 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.284492 4227 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:58:50.284677 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.284567 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:58:50.311920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.311756 4227 generic.go:356] "Generic (PLEG): container finished" podID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerID="3deb5f5a2782f1a988cf54e2b22e6db4f91f00defd347216bfe0f2721d596683" exitCode=0 Nov 03 20:58:50.311920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.311863 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" event={"ID":"1e756a29-2f53-45a6-910c-e9b5907235d8","Type":"ContainerDied","Data":"3deb5f5a2782f1a988cf54e2b22e6db4f91f00defd347216bfe0f2721d596683"} Nov 03 20:58:50.314601 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.314564 4227 generic.go:356] "Generic (PLEG): container finished" podID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerID="1a33dd01f50d16647487b9cbb673725c2304021151edf078c42242042d13a93b" exitCode=0 Nov 03 20:58:50.314740 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.314685 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" event={"ID":"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc","Type":"ContainerDied","Data":"1a33dd01f50d16647487b9cbb673725c2304021151edf078c42242042d13a93b"} Nov 03 20:58:50.317392 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.316993 4227 generic.go:356] "Generic (PLEG): container finished" podID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerID="5362962e9f11fb11fb9e606c01e7a8d73c7f70385ce01e025e8cf379992e2dd6" exitCode=0 Nov 03 20:58:50.317392 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.317143 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" event={"ID":"402854d1-16ef-4ed1-9b1f-c6643af60b3b","Type":"ContainerDied","Data":"5362962e9f11fb11fb9e606c01e7a8d73c7f70385ce01e025e8cf379992e2dd6"} Nov 03 20:58:50.319951 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.319819 4227 generic.go:356] "Generic (PLEG): container finished" podID="fed04739-53d3-4918-9aff-21d4e40945eb" containerID="ddf2638fd38ede1682a6359028be9dad296db3dda8c7a7246735883b8d515728" exitCode=0 Nov 03 20:58:50.319951 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.319892 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" event={"ID":"fed04739-53d3-4918-9aff-21d4e40945eb","Type":"ContainerDied","Data":"ddf2638fd38ede1682a6359028be9dad296db3dda8c7a7246735883b8d515728"} Nov 03 20:58:50.354917 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.354790 4227 generic.go:356] "Generic (PLEG): container finished" podID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerID="4380218ce1864ba5577416313d5480933dc045baf3c63202f93cba2ee03246cc" exitCode=0 Nov 03 20:58:50.354917 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.354820 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" event={"ID":"2d178850-07af-4e82-858d-b9e4e181b8c5","Type":"ContainerDied","Data":"4380218ce1864ba5577416313d5480933dc045baf3c63202f93cba2ee03246cc"} Nov 03 20:58:50.355278 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.355195 4227 scope.go:122] "RemoveContainer" containerID="0a470c8590929436ff0ce92faecedbb06abaa4ebce7791ea4eb42156c85750f4" Nov 03 20:58:50.447253 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.447229 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:58:50.558899 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.558855 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fed04739-53d3-4918-9aff-21d4e40945eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed04739-53d3-4918-9aff-21d4e40945eb-istio-podinfo\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.559092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.558938 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-certs\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.559092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.559003 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed04739-53d3-4918-9aff-21d4e40945eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed04739-53d3-4918-9aff-21d4e40945eb-istiod-ca-cert\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.559092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.559056 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-istio-token\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-istio-token\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.559092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.559085 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-socket\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.559337 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.559131 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-data\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.561506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.561089 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed04739-53d3-4918-9aff-21d4e40945eb-istiod-ca-cert" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:50.561506 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.561373 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-certs" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.561760 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-kube-api-access-lrdxs\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-kube-api-access-lrdxs\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.561801 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-credential-socket\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.561842 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-envoy\") pod \"fed04739-53d3-4918-9aff-21d4e40945eb\" (UID: \"fed04739-53d3-4918-9aff-21d4e40945eb\") " Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.562643 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.562668 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed04739-53d3-4918-9aff-21d4e40945eb-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.563194 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-data" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.563398 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-credential-socket" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.564350 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.563545 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-socket" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.565198 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.565123 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-istio-token" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.567041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.567020 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fed04739-53d3-4918-9aff-21d4e40945eb-istio-podinfo" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:50.568957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.568345 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-kube-api-access-lrdxs" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "kube-api-access-lrdxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.569878 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.569847 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-envoy" pod "fed04739-53d3-4918-9aff-21d4e40945eb" (UID: "fed04739-53d3-4918-9aff-21d4e40945eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.579674 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.579652 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:58:50.620905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.620752 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:58:50.626765 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.626742 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:58:50.629812 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.629778 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:58:50.664056 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664008 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-certs\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.664287 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664060 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-socket\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.664287 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664108 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-kube-api-access-7rh62\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-kube-api-access-7rh62\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.664287 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664226 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-credential-socket\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.664474 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664440 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-socket" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.664722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664696 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-credential-socket" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.664831 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664808 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e756a29-2f53-45a6-910c-e9b5907235d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e756a29-2f53-45a6-910c-e9b5907235d8-istiod-ca-cert\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.665058 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.664830 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-certs" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.665058 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.665029 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e756a29-2f53-45a6-910c-e9b5907235d8-istiod-ca-cert" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:50.665248 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.665077 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-podinfo\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.665329 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.665309 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-data\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.665593 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.665554 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-data" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.665724 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.665707 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-envoy\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.665869 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.665852 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-token\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-token\") pod \"1e756a29-2f53-45a6-910c-e9b5907235d8\" (UID: \"1e756a29-2f53-45a6-910c-e9b5907235d8\") " Nov 03 20:58:50.666988 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.666968 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667108 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667094 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667293 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667275 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrdxs\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-kube-api-access-lrdxs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667408 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667394 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667548 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667534 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed04739-53d3-4918-9aff-21d4e40945eb-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667642 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667630 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667733 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667721 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667845 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.667981 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.667967 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.668229 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.668061 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e756a29-2f53-45a6-910c-e9b5907235d8-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.668368 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.668352 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed04739-53d3-4918-9aff-21d4e40945eb-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.668518 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.668503 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed04739-53d3-4918-9aff-21d4e40945eb-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.668648 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.668210 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-token" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.679122 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.679099 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-podinfo" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:50.679274 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.679132 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-kube-api-access-7rh62" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "kube-api-access-7rh62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.680379 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.680355 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-envoy" pod "1e756a29-2f53-45a6-910c-e9b5907235d8" (UID: "1e756a29-2f53-45a6-910c-e9b5907235d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.769354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769313 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-data\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.769553 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769369 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-data\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.769553 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769415 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-envoy\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.769874 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769849 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-podinfo\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.769974 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769899 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-token\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-token\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.769974 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769946 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istiod-ca-cert\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.770082 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769989 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-certs\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.770082 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.769999 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-data" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.770082 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.770024 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-podinfo\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.770573 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-certs" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.770653 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-data" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.770678 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istiod-ca-cert" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.770780 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istiod-ca-cert\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.770818 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-certs\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771049 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-certs" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771094 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d178850-07af-4e82-858d-b9e4e181b8c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d178850-07af-4e82-858d-b9e4e181b8c5-istiod-ca-cert\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771208 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-socket\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771278 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-certs\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771281 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istiod-ca-cert" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771305 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-token\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-token\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771355 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-credential-socket\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771404 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-kube-api-access-g7z9h\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-kube-api-access-g7z9h\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771503 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-kube-api-access-rc6d7\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-kube-api-access-rc6d7\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771517 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-socket" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771541 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-kube-api-access-4bznt\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-kube-api-access-4bznt\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771568 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-envoy\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771600 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-data\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771626 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-token\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-token\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771664 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-socket\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771740 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-socket\") pod \"2d178850-07af-4e82-858d-b9e4e181b8c5\" (UID: \"2d178850-07af-4e82-858d-b9e4e181b8c5\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771784 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-podinfo\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771811 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-credential-socket\") pod \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\" (UID: \"402854d1-16ef-4ed1-9b1f-c6643af60b3b\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771857 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-envoy\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771861 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d178850-07af-4e82-858d-b9e4e181b8c5-istiod-ca-cert" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:50.772017 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.771895 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-credential-socket\") pod \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\" (UID: \"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc\") " Nov 03 20:58:50.775425 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.775394 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-token" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.777309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.777284 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-credential-socket" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.777629 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.777602 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-certs" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.777718 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.777611 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-data" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.777979 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.777963 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-credential-socket" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.778292 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.778274 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-socket" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.778769 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.778535 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-socket" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.779010 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.778978 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-credential-socket" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.781207 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781181 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d178850-07af-4e82-858d-b9e4e181b8c5-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781315 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781213 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781393 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781375 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781543 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781523 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781543 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781546 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781732 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781717 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781804 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781743 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rh62\" (UniqueName: \"kubernetes.io/projected/1e756a29-2f53-45a6-910c-e9b5907235d8-kube-api-access-7rh62\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781759 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781855 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781835 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.781953 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781855 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.781872 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782075 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782027 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782075 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782041 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782294 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782380 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782317 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e756a29-2f53-45a6-910c-e9b5907235d8-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782485 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782333 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782543 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782490 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782543 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782510 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782679 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782524 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.782736 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.782683 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.785808 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.785766 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-envoy" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.788207 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.788180 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-envoy" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.788302 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.788246 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-kube-api-access-rc6d7" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "kube-api-access-rc6d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.790116 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.790095 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-podinfo" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:50.795916 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.795885 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-envoy" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:50.796020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.795930 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-kube-api-access-4bznt" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "kube-api-access-4bznt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.796020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.795936 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-token" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.796020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.795972 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-podinfo" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:50.796020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.795995 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-podinfo" pod "2d178850-07af-4e82-858d-b9e4e181b8c5" (UID: "2d178850-07af-4e82-858d-b9e4e181b8c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:50.796020 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.795966 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-token" pod "402854d1-16ef-4ed1-9b1f-c6643af60b3b" (UID: "402854d1-16ef-4ed1-9b1f-c6643af60b3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.797385 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.797362 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-kube-api-access-g7z9h" pod "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" (UID: "91f4bbf3-cfda-43a4-bf24-359a4bca5dcc"). InnerVolumeSpecName "kube-api-access-g7z9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:50.884339 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884248 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6d7\" (UniqueName: \"kubernetes.io/projected/2d178850-07af-4e82-858d-b9e4e181b8c5-kube-api-access-rc6d7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884339 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884284 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bznt\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-kube-api-access-4bznt\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884339 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884302 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884339 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884318 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884339 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884332 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884809 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884345 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884809 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884360 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884809 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884374 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d178850-07af-4e82-858d-b9e4e181b8c5-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884809 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884387 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884809 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884402 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402854d1-16ef-4ed1-9b1f-c6643af60b3b-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.884809 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.884417 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7z9h\" (UniqueName: \"kubernetes.io/projected/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc-kube-api-access-g7z9h\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.901437 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.901409 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b690173-19d4-4bdc-8606-3a144223e616" path="/var/lib/kubelet/pods/4b690173-19d4-4bdc-8606-3a144223e616/volumes" Nov 03 20:58:50.901877 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:50.901862 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" path="/var/lib/kubelet/pods/efaf6ae3-ef78-49b7-8b0c-b6ee2169102c/volumes" Nov 03 20:58:51.363224 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.363183 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" event={"ID":"fed04739-53d3-4918-9aff-21d4e40945eb","Type":"ContainerDied","Data":"21f1659824ad2e7f18da4400f0fd051af4e7e818f1028077969f8a2db69aa13a"} Nov 03 20:58:51.363671 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.363256 4227 scope.go:122] "RemoveContainer" containerID="ddf2638fd38ede1682a6359028be9dad296db3dda8c7a7246735883b8d515728" Nov 03 20:58:51.363671 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.363195 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6" Nov 03 20:58:51.364848 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.364828 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" Nov 03 20:58:51.364957 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.364872 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr" event={"ID":"2d178850-07af-4e82-858d-b9e4e181b8c5","Type":"ContainerDied","Data":"851a777f77c67e522de3f2f5b9d3e5c7f5f58682478f7cb04b123abf8d934665"} Nov 03 20:58:51.366458 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.366439 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" Nov 03 20:58:51.366552 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.366470 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm" event={"ID":"1e756a29-2f53-45a6-910c-e9b5907235d8","Type":"ContainerDied","Data":"da17e961bf88f6cd7a577502b1cbebd365643f9665186134443c3b5fcd770ee3"} Nov 03 20:58:51.367987 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.367966 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" event={"ID":"91f4bbf3-cfda-43a4-bf24-359a4bca5dcc","Type":"ContainerDied","Data":"7a8f34e098daa161f3720e454befa10687c669d800a52380b465e85e18900b54"} Nov 03 20:58:51.368090 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.367971 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb" Nov 03 20:58:51.369504 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.369482 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" event={"ID":"402854d1-16ef-4ed1-9b1f-c6643af60b3b","Type":"ContainerDied","Data":"e219964023eb57111dc819e772f3295c623fdd3d90bfe2240abbc28457427f20"} Nov 03 20:58:51.369592 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.369538 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg" Nov 03 20:58:51.377321 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.377277 4227 scope.go:122] "RemoveContainer" containerID="4380218ce1864ba5577416313d5480933dc045baf3c63202f93cba2ee03246cc" Nov 03 20:58:51.385967 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.385951 4227 scope.go:122] "RemoveContainer" containerID="3deb5f5a2782f1a988cf54e2b22e6db4f91f00defd347216bfe0f2721d596683" Nov 03 20:58:51.394128 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.394109 4227 scope.go:122] "RemoveContainer" containerID="1a33dd01f50d16647487b9cbb673725c2304021151edf078c42242042d13a93b" Nov 03 20:58:51.402320 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.402299 4227 scope.go:122] "RemoveContainer" containerID="5362962e9f11fb11fb9e606c01e7a8d73c7f70385ce01e025e8cf379992e2dd6" Nov 03 20:58:51.518150 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.518121 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb"] Nov 03 20:58:51.521271 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.521249 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hr5cb"] Nov 03 20:58:51.624499 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.624417 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm"] Nov 03 20:58:51.627525 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.627505 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-dsvpm"] Nov 03 20:58:51.728494 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.728470 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6"] Nov 03 20:58:51.732316 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.732293 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-h7pk6"] Nov 03 20:58:51.833882 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.833854 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr"] Nov 03 20:58:51.838045 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.838025 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-xsbnr"] Nov 03 20:58:51.940150 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.940119 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg"] Nov 03 20:58:51.944523 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:51.944503 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-dmxcg"] Nov 03 20:58:52.908915 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:52.908881 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" path="/var/lib/kubelet/pods/1e756a29-2f53-45a6-910c-e9b5907235d8/volumes" Nov 03 20:58:52.909411 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:52.909361 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" path="/var/lib/kubelet/pods/2d178850-07af-4e82-858d-b9e4e181b8c5/volumes" Nov 03 20:58:52.909818 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:52.909803 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" path="/var/lib/kubelet/pods/402854d1-16ef-4ed1-9b1f-c6643af60b3b/volumes" Nov 03 20:58:52.910437 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:52.910421 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" path="/var/lib/kubelet/pods/91f4bbf3-cfda-43a4-bf24-359a4bca5dcc/volumes" Nov 03 20:58:52.910868 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:52.910854 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" path="/var/lib/kubelet/pods/fed04739-53d3-4918-9aff-21d4e40945eb/volumes" Nov 03 20:58:53.085435 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.085410 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:58:53.090006 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.089981 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 20:58:53.193762 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.193737 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk"] Nov 03 20:58:53.194513 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194494 4227 cpu_manager.go:414] "containerMap: removing container" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" Nov 03 20:58:53.194588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194515 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" Nov 03 20:58:53.194588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194534 4227 cpu_manager.go:414] "containerMap: removing container" podUID="59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" containerName="maas-api" Nov 03 20:58:53.194588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194542 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" containerName="maas-api" Nov 03 20:58:53.194588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194555 4227 cpu_manager.go:414] "containerMap: removing container" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" Nov 03 20:58:53.194588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194562 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" Nov 03 20:58:53.194588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194582 4227 cpu_manager.go:414] "containerMap: removing container" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" Nov 03 20:58:53.194588 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194589 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194604 4227 cpu_manager.go:414] "containerMap: removing container" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194611 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194628 4227 cpu_manager.go:414] "containerMap: removing container" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194636 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194647 4227 cpu_manager.go:414] "containerMap: removing container" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194654 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194666 4227 cpu_manager.go:414] "containerMap: removing container" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194673 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194692 4227 cpu_manager.go:414] "containerMap: removing container" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194699 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194719 4227 cpu_manager.go:414] "containerMap: removing container" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194726 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194740 4227 cpu_manager.go:414] "containerMap: removing container" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194747 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194757 4227 cpu_manager.go:414] "containerMap: removing container" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" Nov 03 20:58:53.194838 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194763 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194887 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed04739-53d3-4918-9aff-21d4e40945eb" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194900 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="efaf6ae3-ef78-49b7-8b0c-b6ee2169102c" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194907 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ec80df-fefb-4a10-9874-922d5447874b" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194918 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef43ca2-319b-4b4a-bbc4-4bde9da7c1ed" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194928 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f4bbf3-cfda-43a4-bf24-359a4bca5dcc" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194939 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="b472214b-3588-487d-be6e-2004270492cb" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194948 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ffc7a6-5c6f-42d7-9e1c-61d7f9e6c71d" containerName="maas-api" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194957 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194966 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b690173-19d4-4bdc-8606-3a144223e616" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194974 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e756a29-2f53-45a6-910c-e9b5907235d8" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.194984 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d178850-07af-4e82-858d-b9e4e181b8c5" containerName="istio-proxy" Nov 03 20:58:53.195357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.195037 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="402854d1-16ef-4ed1-9b1f-c6643af60b3b" containerName="istio-proxy" Nov 03 20:58:53.200904 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.200881 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.253248 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.253223 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk"] Nov 03 20:58:53.295092 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.295056 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-211d68bf-serving-cert\"" Nov 03 20:58:53.313734 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.313705 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-metrics-certs\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.313861 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.313766 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad49ee-ca87-4d5a-aebd-528d7e672374-openshift-service-ca\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.313905 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.313864 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4vz\" (UniqueName: \"kubernetes.io/projected/daad49ee-ca87-4d5a-aebd-528d7e672374-kube-api-access-xp4vz\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.313938 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.313920 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.415471 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.415355 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-metrics-certs\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.415621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.415481 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad49ee-ca87-4d5a-aebd-528d7e672374-openshift-service-ca\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.415621 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.415533 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4vz\" (UniqueName: \"kubernetes.io/projected/daad49ee-ca87-4d5a-aebd-528d7e672374-kube-api-access-xp4vz\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.415737 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.415618 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.415799 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:53.415777 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-serving-cert: secret "maas-api-e2e-mt-211d68bf-serving-cert" not found Nov 03 20:58:53.415856 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:53.415845 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls podName:daad49ee-ca87-4d5a-aebd-528d7e672374 nodeName:}" failed. No retries permitted until 2026-11-03 20:58:53.915828164 +0000 UTC m=+1801.152806017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls") pod "maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" (UID: "daad49ee-ca87-4d5a-aebd-528d7e672374") : secret "maas-api-e2e-mt-211d68bf-serving-cert" not found Nov 03 20:58:53.416284 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.416264 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad49ee-ca87-4d5a-aebd-528d7e672374-openshift-service-ca\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.418243 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.418220 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-metrics-certs\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.487200 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.486981 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4vz\" (UniqueName: \"kubernetes.io/projected/daad49ee-ca87-4d5a-aebd-528d7e672374-kube-api-access-xp4vz\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.928539 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.928505 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:53.930940 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:53.930911 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls\") pod \"maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:54.118877 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:54.118840 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:54.493584 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:54.493551 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk"] Nov 03 20:58:54.499590 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:58:54.499560 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaad49ee_ca87_4d5a_aebd_528d7e672374.slice/crio-bb59125c0490bcab3e5596a86eae662fcb757947e06ac1085c24df8ca0866693 WatchSource:0}: Error finding container bb59125c0490bcab3e5596a86eae662fcb757947e06ac1085c24df8ca0866693: Status 404 returned error can't find the container with id bb59125c0490bcab3e5596a86eae662fcb757947e06ac1085c24df8ca0866693 Nov 03 20:58:54.501264 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:54.501245 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:58:55.403309 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:55.403268 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" event={"ID":"daad49ee-ca87-4d5a-aebd-528d7e672374","Type":"ContainerStarted","Data":"bb59125c0490bcab3e5596a86eae662fcb757947e06ac1085c24df8ca0866693"} Nov 03 20:58:56.081205 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:58:56.081139 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:58:56.413890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:56.413846 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" event={"ID":"daad49ee-ca87-4d5a-aebd-528d7e672374","Type":"ContainerStarted","Data":"62a6351fcfbdb9c82d078216ee15f86306f1022f46797bacaf69fc4cec40a839"} Nov 03 20:58:56.414388 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:56.414012 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:58:56.522055 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:58:56.521994 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" podStartSLOduration=2.522433501 podStartE2EDuration="3.521980206s" totalImagesPullingTime="999.546705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:56.519563822 +0000 UTC m=+1803.756541696" watchObservedRunningTime="2026-11-03 20:58:56.521980206 +0000 UTC m=+1803.758958118" Nov 03 20:59:02.426047 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:02.426019 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 20:59:12.239252 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.239218 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z"] Nov 03 20:59:12.243357 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.243338 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.332944 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.332916 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24wc\" (UniqueName: \"kubernetes.io/projected/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-kube-api-access-l24wc\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.333128 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.332972 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.333128 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.333106 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.333348 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.333328 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.339188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.339134 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-b6d800-serving-cert\"" Nov 03 20:59:12.342358 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.342334 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z"] Nov 03 20:59:12.434696 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.434663 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.434849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.434789 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24wc\" (UniqueName: \"kubernetes.io/projected/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-kube-api-access-l24wc\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.434849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.434817 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.434849 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.434843 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.435014 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:12.434982 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-serving-cert: secret "maas-api-e2e-shared-a-w5-b6d800-serving-cert" not found Nov 03 20:59:12.435064 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:12.435057 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls podName:fdea3f24-37ab-4bff-ad3f-a4b221fa37f0 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:12.935033823 +0000 UTC m=+1820.172011675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls") pod "maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" (UID: "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0") : secret "maas-api-e2e-shared-a-w5-b6d800-serving-cert" not found Nov 03 20:59:12.435530 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.435511 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.437096 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.437073 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.539235 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.521447 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24wc\" (UniqueName: \"kubernetes.io/projected/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-kube-api-access-l24wc\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.980865 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.980826 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:12.983306 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:12.983277 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:13.156674 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:13.156626 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:13.603013 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:13.602968 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z"] Nov 03 20:59:13.611016 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:59:13.610987 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdea3f24_37ab_4bff_ad3f_a4b221fa37f0.slice/crio-68e107fca0df24379e1d8d83cf0320c8e9980c6b65dea7482d3e25801653a634 WatchSource:0}: Error finding container 68e107fca0df24379e1d8d83cf0320c8e9980c6b65dea7482d3e25801653a634: Status 404 returned error can't find the container with id 68e107fca0df24379e1d8d83cf0320c8e9980c6b65dea7482d3e25801653a634 Nov 03 20:59:13.686495 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:13.686460 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" event={"ID":"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0","Type":"ContainerStarted","Data":"68e107fca0df24379e1d8d83cf0320c8e9980c6b65dea7482d3e25801653a634"} Nov 03 20:59:14.696279 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:14.696240 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" event={"ID":"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0","Type":"ContainerStarted","Data":"f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7"} Nov 03 20:59:14.696692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:14.696373 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:14.836346 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:14.836248 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" podStartSLOduration=1.9000103830000001 podStartE2EDuration="2.836233514s" totalImagesPullingTime="936.223131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:14.833614311 +0000 UTC m=+1822.070592185" watchObservedRunningTime="2026-11-03 20:59:14.836233514 +0000 UTC m=+1822.073211387" Nov 03 20:59:20.708084 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:20.708052 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 20:59:26.083650 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:26.083607 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 20:59:26.588637 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.588600 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp"] Nov 03 20:59:26.593021 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.592998 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.642216 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.642182 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1e862b-serving-cert\"" Nov 03 20:59:26.691890 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.691856 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp"] Nov 03 20:59:26.702722 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.702686 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.702901 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.702727 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95n6q\" (UniqueName: \"kubernetes.io/projected/ee287795-b1a2-4763-8eec-f7ef207e277b-kube-api-access-95n6q\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.702901 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.702819 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.703004 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.702983 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee287795-b1a2-4763-8eec-f7ef207e277b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.804412 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.804373 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.804612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.804418 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95n6q\" (UniqueName: \"kubernetes.io/projected/ee287795-b1a2-4763-8eec-f7ef207e277b-kube-api-access-95n6q\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.804612 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.804492 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.804754 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:26.804663 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-serving-cert: secret "maas-api-e2e-shared-b-w3-1e862b-serving-cert" not found Nov 03 20:59:26.804754 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:26.804746 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls podName:ee287795-b1a2-4763-8eec-f7ef207e277b nodeName:}" failed. No retries permitted until 2026-11-03 20:59:27.30472588 +0000 UTC m=+1834.541703950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" (UID: "ee287795-b1a2-4763-8eec-f7ef207e277b") : secret "maas-api-e2e-shared-b-w3-1e862b-serving-cert" not found Nov 03 20:59:26.804873 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.804664 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee287795-b1a2-4763-8eec-f7ef207e277b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.805709 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.805681 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee287795-b1a2-4763-8eec-f7ef207e277b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.807239 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.807213 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:26.959920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:26.959884 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95n6q\" (UniqueName: \"kubernetes.io/projected/ee287795-b1a2-4763-8eec-f7ef207e277b-kube-api-access-95n6q\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:27.332257 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:27.332036 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:27.335977 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:27.335950 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:27.506927 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:27.506894 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:27.883787 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:27.883755 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp"] Nov 03 20:59:27.889607 ip-10-0-134-71 kubenswrapper[4227]: W1103 20:59:27.889574 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee287795_b1a2_4763_8eec_f7ef207e277b.slice/crio-94aff520f5e16168c93857ed6b046ccdc5e4e27e03a1c7afabeff8330f309880 WatchSource:0}: Error finding container 94aff520f5e16168c93857ed6b046ccdc5e4e27e03a1c7afabeff8330f309880: Status 404 returned error can't find the container with id 94aff520f5e16168c93857ed6b046ccdc5e4e27e03a1c7afabeff8330f309880 Nov 03 20:59:28.050452 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.050415 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk"] Nov 03 20:59:28.050664 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.050621 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" containerID="cri-o://5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31" gracePeriod=30 Nov 03 20:59:28.097721 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.097683 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg"] Nov 03 20:59:28.097947 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.097921 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" containerID="cri-o://35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb" gracePeriod=30 Nov 03 20:59:28.100749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.100718 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8"] Nov 03 20:59:28.100955 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.100927 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" containerID="cri-o://8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c" gracePeriod=30 Nov 03 20:59:28.104354 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.104331 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj"] Nov 03 20:59:28.104560 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.104538 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" containerID="cri-o://3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d" gracePeriod=30 Nov 03 20:59:28.813213 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:28.812691 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" event={"ID":"ee287795-b1a2-4763-8eec-f7ef207e277b","Type":"ContainerStarted","Data":"94aff520f5e16168c93857ed6b046ccdc5e4e27e03a1c7afabeff8330f309880"} Nov 03 20:59:29.820479 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:29.820445 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" event={"ID":"ee287795-b1a2-4763-8eec-f7ef207e277b","Type":"ContainerStarted","Data":"c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3"} Nov 03 20:59:29.820850 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:29.820517 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:29.944494 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:29.944440 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" podStartSLOduration=3.007171865 podStartE2EDuration="3.944416629s" totalImagesPullingTime="937.244764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:29.941946309 +0000 UTC m=+1837.178924185" watchObservedRunningTime="2026-11-03 20:59:29.944416629 +0000 UTC m=+1837.181394497" Nov 03 20:59:33.394144 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.394114 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:59:33.426930 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.426902 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:59:33.468582 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.468547 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:59:33.470811 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.470783 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-envoy\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.470952 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.470926 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f98bb0-8ca4-4687-826c-ed9164a46130-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f98bb0-8ca4-4687-826c-ed9164a46130-istiod-ca-cert\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471022 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.470966 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-socket\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471138 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471115 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-kube-api-access-trgcm\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-kube-api-access-trgcm\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471277 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471152 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-data\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471277 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471209 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-certs\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471277 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471270 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-credential-socket\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471497 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471300 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-token\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-token\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471497 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471331 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-podinfo\") pod \"44f98bb0-8ca4-4687-826c-ed9164a46130\" (UID: \"44f98bb0-8ca4-4687-826c-ed9164a46130\") " Nov 03 20:59:33.471497 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471470 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f98bb0-8ca4-4687-826c-ed9164a46130-istiod-ca-cert" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.471749 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471725 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-data" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.471973 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471954 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f98bb0-8ca4-4687-826c-ed9164a46130-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.472041 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.471978 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.472427 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.472184 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-socket" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.472427 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.472362 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-credential-socket" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.472427 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.472391 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-certs" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.473298 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.473267 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-envoy" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.474522 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.474494 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-kube-api-access-trgcm" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "kube-api-access-trgcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.474750 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.474729 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-token" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.474946 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.474931 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-podinfo" pod "44f98bb0-8ca4-4687-826c-ed9164a46130" (UID: "44f98bb0-8ca4-4687-826c-ed9164a46130"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.507068 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.507029 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:59:33.572973 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.572938 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istiod-ca-cert\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.572978 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-podinfo\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573054 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c871eb20-8eba-4e0a-9271-5e182783dcd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871eb20-8eba-4e0a-9271-5e182783dcd3-istiod-ca-cert\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573085 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-envoy\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573188 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573137 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-token\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-token\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573439 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573211 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-podinfo\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573439 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573244 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-envoy\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573439 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573343 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-kube-api-access-vzrjc\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-kube-api-access-vzrjc\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573605 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573447 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-socket\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573605 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573491 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-credential-socket\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573605 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573527 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-data\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.573605 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573554 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-socket\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573605 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573586 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-credential-socket\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573626 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-kube-api-access-pbjft\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-kube-api-access-pbjft\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573673 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-token\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-token\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573727 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-certs\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573769 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-data\") pod \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\" (UID: \"6a533677-a8a8-4a9b-b2ed-01313479f2f0\") " Nov 03 20:59:33.573859 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.573807 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-certs\") pod \"c871eb20-8eba-4e0a-9271-5e182783dcd3\" (UID: \"c871eb20-8eba-4e0a-9271-5e182783dcd3\") " Nov 03 20:59:33.574610 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574586 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trgcm\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-kube-api-access-trgcm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.574742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574614 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.574742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574631 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.574742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574646 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.574742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574660 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.574742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574657 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c871eb20-8eba-4e0a-9271-5e182783dcd3-istiod-ca-cert" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.574742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574673 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.574742 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574696 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f98bb0-8ca4-4687-826c-ed9164a46130-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.575138 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574872 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-certs" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.575138 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.574956 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-credential-socket" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.575286 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.575138 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-socket" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.575433 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.575409 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-socket" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.577263 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.577194 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-certs" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.577397 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.577376 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istiod-ca-cert" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.577808 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.577744 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-data" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.577808 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.577791 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-credential-socket" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.579304 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.579275 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-data" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.580080 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.579849 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-podinfo" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.580080 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.579859 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-podinfo" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.580382 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.580349 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-kube-api-access-vzrjc" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "kube-api-access-vzrjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.580870 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.580833 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-token" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.582034 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.581818 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-envoy" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.582034 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.581839 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-envoy" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.583538 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.582731 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-kube-api-access-pbjft" pod "6a533677-a8a8-4a9b-b2ed-01313479f2f0" (UID: "6a533677-a8a8-4a9b-b2ed-01313479f2f0"). InnerVolumeSpecName "kube-api-access-pbjft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.586535 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.586509 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-token" pod "c871eb20-8eba-4e0a-9271-5e182783dcd3" (UID: "c871eb20-8eba-4e0a-9271-5e182783dcd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.675608 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675563 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-istio-token\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-istio-token\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.675822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675645 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-credential-socket\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.675822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675703 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f016ad-0799-4093-b879-749bf738eea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f016ad-0799-4093-b879-749bf738eea5-istiod-ca-cert\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.675822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675751 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-socket\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.675822 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675806 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-envoy\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.676061 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675922 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-kube-api-access-t2dfn\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-kube-api-access-t2dfn\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.676061 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675973 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5f016ad-0799-4093-b879-749bf738eea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f016ad-0799-4093-b879-749bf738eea5-istio-podinfo\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.676061 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676003 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-data\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.676268 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676127 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-certs\") pod \"a5f016ad-0799-4093-b879-749bf738eea5\" (UID: \"a5f016ad-0799-4093-b879-749bf738eea5\") " Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676706 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676726 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676736 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676745 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrjc\" (UniqueName: \"kubernetes.io/projected/c871eb20-8eba-4e0a-9271-5e182783dcd3-kube-api-access-vzrjc\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676756 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676765 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676775 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676784 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676803 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676823 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjft\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-kube-api-access-pbjft\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676837 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676852 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676865 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676880 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871eb20-8eba-4e0a-9271-5e182783dcd3-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676892 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676907 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676920 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871eb20-8eba-4e0a-9271-5e182783dcd3-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.676933 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a533677-a8a8-4a9b-b2ed-01313479f2f0-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.675996 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-credential-socket" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.677316 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-certs" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.677448 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-socket" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.677507 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-data" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.677706 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.677677 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f016ad-0799-4093-b879-749bf738eea5-istiod-ca-cert" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.679019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.678982 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-envoy" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.679569 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.679538 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f016ad-0799-4093-b879-749bf738eea5-istio-podinfo" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.680435 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.680402 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-istio-token" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.701220 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.688785 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-kube-api-access-t2dfn" pod "a5f016ad-0799-4093-b879-749bf738eea5" (UID: "a5f016ad-0799-4093-b879-749bf738eea5"). InnerVolumeSpecName "kube-api-access-t2dfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781510 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dfn\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-kube-api-access-t2dfn\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781549 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f016ad-0799-4093-b879-749bf738eea5-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781565 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781579 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781593 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f016ad-0799-4093-b879-749bf738eea5-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781606 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781619 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f016ad-0799-4093-b879-749bf738eea5-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781634 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.784187 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.781648 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f016ad-0799-4093-b879-749bf738eea5-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.934440 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.934347 4227 generic.go:356] "Generic (PLEG): container finished" podID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerID="35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb" exitCode=0 Nov 03 20:59:33.934440 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.934414 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" Nov 03 20:59:33.934675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.934435 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" event={"ID":"6a533677-a8a8-4a9b-b2ed-01313479f2f0","Type":"ContainerDied","Data":"35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb"} Nov 03 20:59:33.934675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.934481 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg" event={"ID":"6a533677-a8a8-4a9b-b2ed-01313479f2f0","Type":"ContainerDied","Data":"018a46fe55a72ccfed8462de02b5ef00c3e82761af8af55ad0653110de004f61"} Nov 03 20:59:33.934675 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.934502 4227 scope.go:122] "RemoveContainer" containerID="35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb" Nov 03 20:59:33.938863 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.938838 4227 generic.go:356] "Generic (PLEG): container finished" podID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerID="5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31" exitCode=0 Nov 03 20:59:33.938990 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.938913 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" event={"ID":"44f98bb0-8ca4-4687-826c-ed9164a46130","Type":"ContainerDied","Data":"5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31"} Nov 03 20:59:33.938990 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.938979 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" event={"ID":"44f98bb0-8ca4-4687-826c-ed9164a46130","Type":"ContainerDied","Data":"fd2580f5ee350c7f9671c3de459099651f022c6a9046374b76f2e941a14bc2ec"} Nov 03 20:59:33.939101 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.939037 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk" Nov 03 20:59:33.940744 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.940718 4227 generic.go:356] "Generic (PLEG): container finished" podID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerID="8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c" exitCode=0 Nov 03 20:59:33.944666 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.940928 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" event={"ID":"c871eb20-8eba-4e0a-9271-5e182783dcd3","Type":"ContainerDied","Data":"8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c"} Nov 03 20:59:33.944666 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.940976 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" event={"ID":"c871eb20-8eba-4e0a-9271-5e182783dcd3","Type":"ContainerDied","Data":"8c0efed8677a2b522200b6a64d58fd4b6bb2e89738da88e6316a567fdc2851ec"} Nov 03 20:59:33.944666 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.941056 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8" Nov 03 20:59:33.951794 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.951710 4227 generic.go:356] "Generic (PLEG): container finished" podID="a5f016ad-0799-4093-b879-749bf738eea5" containerID="3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d" exitCode=0 Nov 03 20:59:33.951920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.951801 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" event={"ID":"a5f016ad-0799-4093-b879-749bf738eea5","Type":"ContainerDied","Data":"3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d"} Nov 03 20:59:33.951920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.951844 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" event={"ID":"a5f016ad-0799-4093-b879-749bf738eea5","Type":"ContainerDied","Data":"259238a5e0eb9831b71fc61bfeb9aed056d5df058c5d4a75a015c72e6950b972"} Nov 03 20:59:33.951920 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.951918 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj" Nov 03 20:59:33.967961 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.967940 4227 scope.go:122] "RemoveContainer" containerID="35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb" Nov 03 20:59:33.968652 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:33.968622 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb\": container with ID starting with 35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb not found: ID does not exist" containerID="35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb" Nov 03 20:59:33.968739 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.968663 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb"} err="failed to get container status \"35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb\": rpc error: code = NotFound desc = could not find container \"35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb\": container with ID starting with 35f6e2ff04222a3edfa056cf592c40a73c4006b0fa9222a2d4db581febbfddeb not found: ID does not exist" Nov 03 20:59:33.968739 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.968686 4227 scope.go:122] "RemoveContainer" containerID="5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31" Nov 03 20:59:33.991142 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.990330 4227 scope.go:122] "RemoveContainer" containerID="5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31" Nov 03 20:59:33.992514 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:33.992483 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31\": container with ID starting with 5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31 not found: ID does not exist" containerID="5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31" Nov 03 20:59:33.992616 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.992524 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31"} err="failed to get container status \"5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31\": rpc error: code = NotFound desc = could not find container \"5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31\": container with ID starting with 5dfd754bb38149894000a8a2bea19aab96eecbc1580e4c364b7308478be47b31 not found: ID does not exist" Nov 03 20:59:33.992616 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:33.992550 4227 scope.go:122] "RemoveContainer" containerID="8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c" Nov 03 20:59:34.003311 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.003290 4227 scope.go:122] "RemoveContainer" containerID="8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c" Nov 03 20:59:34.003686 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:34.003657 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c\": container with ID starting with 8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c not found: ID does not exist" containerID="8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c" Nov 03 20:59:34.003756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.003693 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c"} err="failed to get container status \"8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c\": rpc error: code = NotFound desc = could not find container \"8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c\": container with ID starting with 8b5abfaccd89a7cae25238492d162aba7e2be13b7affc5f254f3863eac411c1c not found: ID does not exist" Nov 03 20:59:34.003756 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.003711 4227 scope.go:122] "RemoveContainer" containerID="3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d" Nov 03 20:59:34.020752 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.020629 4227 scope.go:122] "RemoveContainer" containerID="3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d" Nov 03 20:59:34.022381 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:34.022351 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d\": container with ID starting with 3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d not found: ID does not exist" containerID="3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d" Nov 03 20:59:34.022524 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.022390 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d"} err="failed to get container status \"3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d\": rpc error: code = NotFound desc = could not find container \"3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d\": container with ID starting with 3263a870295057ea9b750594f5c9507b9be912884133951fb0998fad46b6013d not found: ID does not exist" Nov 03 20:59:34.149316 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.149248 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8"] Nov 03 20:59:34.155844 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.155815 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-n7gk8"] Nov 03 20:59:34.256019 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.255933 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg"] Nov 03 20:59:34.261773 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.261740 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-8m2kg"] Nov 03 20:59:34.361515 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.361479 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj"] Nov 03 20:59:34.364692 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.364666 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-dlgvj"] Nov 03 20:59:34.468823 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.468788 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk"] Nov 03 20:59:34.473307 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.473280 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-4xsfk"] Nov 03 20:59:34.900241 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.900206 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" path="/var/lib/kubelet/pods/44f98bb0-8ca4-4687-826c-ed9164a46130/volumes" Nov 03 20:59:34.900791 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.900775 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" path="/var/lib/kubelet/pods/6a533677-a8a8-4a9b-b2ed-01313479f2f0/volumes" Nov 03 20:59:34.901289 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.901274 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f016ad-0799-4093-b879-749bf738eea5" path="/var/lib/kubelet/pods/a5f016ad-0799-4093-b879-749bf738eea5/volumes" Nov 03 20:59:34.902136 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:34.902109 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" path="/var/lib/kubelet/pods/c871eb20-8eba-4e0a-9271-5e182783dcd3/volumes" Nov 03 20:59:35.848860 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:35.848831 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 20:59:53.622237 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:53.621297 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.753191 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:53.752780 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.758218 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:53.753921 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.770467 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:53.770413 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:53.771616 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:53.771589 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.069112 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.069058 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.155185 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.154374 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.243359 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.243310 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.463292 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.463243 4227 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 20:59:54.757224 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.756258 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:59:54.757224 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.757037 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:59:54.770217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.768257 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:54.770217 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.768634 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:59:54.857847 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:54.855485 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 20:59:55.040472 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:55.040379 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 20:59:55.096266 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:55.096217 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:55.097214 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:55.097195 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:59:55.189183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:55.162291 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:55.771183 ip-10-0-134-71 kubenswrapper[4227]: I1103 20:59:55.770656 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 20:59:56.135832 ip-10-0-134-71 kubenswrapper[4227]: E1103 20:59:56.135714 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:00:00.224596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.224554 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz"] Nov 03 21:00:00.239108 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.239070 4227 cpu_manager.go:414] "containerMap: removing container" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" Nov 03 21:00:00.239378 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.239360 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" Nov 03 21:00:00.239527 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.239511 4227 cpu_manager.go:414] "containerMap: removing container" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" Nov 03 21:00:00.239613 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.239600 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" Nov 03 21:00:00.239735 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.239720 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" Nov 03 21:00:00.239837 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.239824 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" Nov 03 21:00:00.239945 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.239931 4227 cpu_manager.go:414] "containerMap: removing container" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" Nov 03 21:00:00.240037 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.240025 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" Nov 03 21:00:00.240295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.240279 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f98bb0-8ca4-4687-826c-ed9164a46130" containerName="istio-proxy" Nov 03 21:00:00.240406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.240387 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c871eb20-8eba-4e0a-9271-5e182783dcd3" containerName="istio-proxy" Nov 03 21:00:00.240501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.240491 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f016ad-0799-4093-b879-749bf738eea5" containerName="istio-proxy" Nov 03 21:00:00.240589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.240577 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a533677-a8a8-4a9b-b2ed-01313479f2f0" containerName="istio-proxy" Nov 03 21:00:00.274217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.271979 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78"] Nov 03 21:00:00.274217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.272279 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.281957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.278217 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:00.328958 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.328928 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:00:00.375677 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.375639 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz"] Nov 03 21:00:00.389736 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.385766 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78"] Nov 03 21:00:00.459587 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.459544 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nhg\" (UniqueName: \"kubernetes.io/projected/ab570580-6267-4e2b-a2fe-444e8116a935-kube-api-access-b9nhg\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:00.459780 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.459631 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab570580-6267-4e2b-a2fe-444e8116a935-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:00.459780 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.459728 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfd7w\" (UniqueName: \"kubernetes.io/projected/a47a13bb-cdc0-43b3-9766-9cff9d518d40-kube-api-access-qfd7w\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.459901 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.459795 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a13bb-cdc0-43b3-9766-9cff9d518d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.560651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.560565 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfd7w\" (UniqueName: \"kubernetes.io/projected/a47a13bb-cdc0-43b3-9766-9cff9d518d40-kube-api-access-qfd7w\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.560651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.560641 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a13bb-cdc0-43b3-9766-9cff9d518d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.560914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.560763 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nhg\" (UniqueName: \"kubernetes.io/projected/ab570580-6267-4e2b-a2fe-444e8116a935-kube-api-access-b9nhg\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:00.560914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.560833 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab570580-6267-4e2b-a2fe-444e8116a935-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:00.561504 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.561475 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab570580-6267-4e2b-a2fe-444e8116a935-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:00.561616 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.561555 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a13bb-cdc0-43b3-9766-9cff9d518d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.615111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.615082 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nhg\" (UniqueName: \"kubernetes.io/projected/ab570580-6267-4e2b-a2fe-444e8116a935-kube-api-access-b9nhg\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:00.615284 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.615132 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfd7w\" (UniqueName: \"kubernetes.io/projected/a47a13bb-cdc0-43b3-9766-9cff9d518d40-kube-api-access-qfd7w\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.899344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.898620 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:00.913682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:00.913498 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:01.571214 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:01.570997 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz"] Nov 03 21:00:01.776949 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:01.776898 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78"] Nov 03 21:00:01.920237 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:01.920176 4227 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab570580_6267_4e2b_a2fe_444e8116a935.slice/crio-b7d2bce228a4be8cbe29ff4a63fb90742fba6b8cddbd7df111d06e4f1c29b7c1\": RecentStats: unable to find data in memory cache]" Nov 03 21:00:02.499364 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.499320 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:00:02.499564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.499385 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:02.618793 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.618700 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" event={"ID":"ab570580-6267-4e2b-a2fe-444e8116a935","Type":"ContainerStarted","Data":"3d878b5022855344dc0311c9e3174ba5221bb1db913466bbe24f8d2c209feb1f"} Nov 03 21:00:02.618793 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.618778 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" event={"ID":"ab570580-6267-4e2b-a2fe-444e8116a935","Type":"ContainerStarted","Data":"b7d2bce228a4be8cbe29ff4a63fb90742fba6b8cddbd7df111d06e4f1c29b7c1"} Nov 03 21:00:02.621863 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.621832 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" event={"ID":"a47a13bb-cdc0-43b3-9766-9cff9d518d40","Type":"ContainerStarted","Data":"4d2e4b1dbaa6cfc8404e8e8c18060a2d197c9bdd31d3d21feaa9da2299dd1f4d"} Nov 03 21:00:02.621943 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.621893 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" event={"ID":"a47a13bb-cdc0-43b3-9766-9cff9d518d40","Type":"ContainerStarted","Data":"91a1874880e70280f26f988b0371c61361d8374d584108799604b2f5fb290bb6"} Nov 03 21:00:02.729920 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.729851 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" podStartSLOduration=2.729833047 podStartE2EDuration="2.729833047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.727517509 +0000 UTC m=+1869.964495398" watchObservedRunningTime="2026-11-03 21:00:02.729833047 +0000 UTC m=+1869.966810922" Nov 03 21:00:02.837625 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:02.837560 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" podStartSLOduration=2.837541411 podStartE2EDuration="2.837541411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.833016142 +0000 UTC m=+1870.069994014" watchObservedRunningTime="2026-11-03 21:00:02.837541411 +0000 UTC m=+1870.074519286" Nov 03 21:00:03.630096 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.630061 4227 generic.go:356] "Generic (PLEG): container finished" podID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerID="15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548" exitCode=0 Nov 03 21:00:03.630590 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.630139 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" event={"ID":"ddbea0aa-e253-4668-bdb9-faa486867e9c","Type":"ContainerDied","Data":"15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548"} Nov 03 21:00:03.630675 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.630657 4227 scope.go:122] "RemoveContainer" containerID="15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548" Nov 03 21:00:03.632659 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.632627 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:03.632766 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.632705 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:03.632766 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.632735 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:03.633440 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.633346 4227 generic.go:356] "Generic (PLEG): container finished" podID="ab570580-6267-4e2b-a2fe-444e8116a935" containerID="3d878b5022855344dc0311c9e3174ba5221bb1db913466bbe24f8d2c209feb1f" exitCode=60 Nov 03 21:00:03.633512 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.633445 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" event={"ID":"ab570580-6267-4e2b-a2fe-444e8116a935","Type":"ContainerDied","Data":"3d878b5022855344dc0311c9e3174ba5221bb1db913466bbe24f8d2c209feb1f"} Nov 03 21:00:03.633681 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.633663 4227 scope.go:122] "RemoveContainer" containerID="3d878b5022855344dc0311c9e3174ba5221bb1db913466bbe24f8d2c209feb1f" Nov 03 21:00:03.635016 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.634994 4227 generic.go:356] "Generic (PLEG): container finished" podID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerID="4d2e4b1dbaa6cfc8404e8e8c18060a2d197c9bdd31d3d21feaa9da2299dd1f4d" exitCode=60 Nov 03 21:00:03.635123 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.635021 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" event={"ID":"a47a13bb-cdc0-43b3-9766-9cff9d518d40","Type":"ContainerDied","Data":"4d2e4b1dbaa6cfc8404e8e8c18060a2d197c9bdd31d3d21feaa9da2299dd1f4d"} Nov 03 21:00:03.635304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:03.635284 4227 scope.go:122] "RemoveContainer" containerID="4d2e4b1dbaa6cfc8404e8e8c18060a2d197c9bdd31d3d21feaa9da2299dd1f4d" Nov 03 21:00:04.643209 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:04.643148 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" event={"ID":"ddbea0aa-e253-4668-bdb9-faa486867e9c","Type":"ContainerStarted","Data":"a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d"} Nov 03 21:00:04.645279 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:04.645249 4227 generic.go:356] "Generic (PLEG): container finished" podID="ab570580-6267-4e2b-a2fe-444e8116a935" containerID="7d7a68093ceb1d9061eadec33671c2aa7c886673545e9b9bf5ed8bc55c67c3e9" exitCode=60 Nov 03 21:00:04.645422 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:04.645328 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" event={"ID":"ab570580-6267-4e2b-a2fe-444e8116a935","Type":"ContainerDied","Data":"7d7a68093ceb1d9061eadec33671c2aa7c886673545e9b9bf5ed8bc55c67c3e9"} Nov 03 21:00:04.645422 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:04.645384 4227 scope.go:122] "RemoveContainer" containerID="3d878b5022855344dc0311c9e3174ba5221bb1db913466bbe24f8d2c209feb1f" Nov 03 21:00:04.645667 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:04.645653 4227 scope.go:122] "RemoveContainer" containerID="7d7a68093ceb1d9061eadec33671c2aa7c886673545e9b9bf5ed8bc55c67c3e9" Nov 03 21:00:04.646000 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:04.645969 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78_odh-ai-gateway-infra(ab570580-6267-4e2b-a2fe-444e8116a935)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" Nov 03 21:00:04.659243 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:04.659115 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" event={"ID":"a47a13bb-cdc0-43b3-9766-9cff9d518d40","Type":"ContainerStarted","Data":"3a18672eb990e01fb1a35eac6da9a6331407a202ee5dc046feb7d4d62e1cd703"} Nov 03 21:00:05.362248 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.358576 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 21:00:05.362248 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.358711 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 21:00:05.362248 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.358754 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 21:00:05.683181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.682734 4227 scope.go:122] "RemoveContainer" containerID="7d7a68093ceb1d9061eadec33671c2aa7c886673545e9b9bf5ed8bc55c67c3e9" Nov 03 21:00:05.683181 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:05.683091 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78_odh-ai-gateway-infra(ab570580-6267-4e2b-a2fe-444e8116a935)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" Nov 03 21:00:05.684698 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.684668 4227 generic.go:356] "Generic (PLEG): container finished" podID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerID="3a18672eb990e01fb1a35eac6da9a6331407a202ee5dc046feb7d4d62e1cd703" exitCode=60 Nov 03 21:00:05.684837 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.684754 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" event={"ID":"a47a13bb-cdc0-43b3-9766-9cff9d518d40","Type":"ContainerDied","Data":"3a18672eb990e01fb1a35eac6da9a6331407a202ee5dc046feb7d4d62e1cd703"} Nov 03 21:00:05.684837 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.684822 4227 scope.go:122] "RemoveContainer" containerID="4d2e4b1dbaa6cfc8404e8e8c18060a2d197c9bdd31d3d21feaa9da2299dd1f4d" Nov 03 21:00:05.685042 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:05.685024 4227 scope.go:122] "RemoveContainer" containerID="3a18672eb990e01fb1a35eac6da9a6331407a202ee5dc046feb7d4d62e1cd703" Nov 03 21:00:05.685392 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:05.685370 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz_odh-ai-gateway-infra(a47a13bb-cdc0-43b3-9766-9cff9d518d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" Nov 03 21:00:06.358543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:06.358508 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 21:00:06.358858 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:06.358816 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 21:00:06.787198 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:06.787137 4227 scope.go:122] "RemoveContainer" containerID="3a18672eb990e01fb1a35eac6da9a6331407a202ee5dc046feb7d4d62e1cd703" Nov 03 21:00:06.788529 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:06.788482 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz_odh-ai-gateway-infra(a47a13bb-cdc0-43b3-9766-9cff9d518d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" Nov 03 21:00:07.358182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:07.355529 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 21:00:07.358182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:07.355606 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 21:00:08.368184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:08.360252 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 21:00:08.368184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:08.360326 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 21:00:09.370184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:09.356526 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 21:00:09.370184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:09.356622 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 21:00:10.371845 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.371802 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6"] Nov 03 21:00:10.384522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.379577 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.422389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.422340 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6"] Nov 03 21:00:10.477184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.476775 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 21:00:10.503239 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.477813 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-46ca0b\"" Nov 03 21:00:10.529185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.481719 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 21:00:10.529557 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.529533 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 21:00:10.529826 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.529809 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6"] Nov 03 21:00:10.535179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.526217 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvn4t\" (UniqueName: \"kubernetes.io/projected/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-kube-api-access-wvn4t\") pod \"payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.535179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.533522 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.554207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.481979 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-46ca0b-dockercfg-czfhx\"" Nov 03 21:00:10.608189 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.604201 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6"] Nov 03 21:00:10.608189 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.604323 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:10.664242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.635102 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvn4t\" (UniqueName: \"kubernetes.io/projected/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-kube-api-access-wvn4t\") pod \"payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.664242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.644539 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.664242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.645611 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.720372 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.719972 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvn4t\" (UniqueName: \"kubernetes.io/projected/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-kube-api-access-wvn4t\") pod \"payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.739942 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.739914 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:10.745622 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.745591 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4sz\" (UniqueName: \"kubernetes.io/projected/6ec6f49d-4114-4945-929c-f5a6b4c70679-kube-api-access-qf4sz\") pod \"payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:10.745762 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.745747 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec6f49d-4114-4945-929c-f5a6b4c70679-plugins-config-volume\") pod \"payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:10.846729 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.846690 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4sz\" (UniqueName: \"kubernetes.io/projected/6ec6f49d-4114-4945-929c-f5a6b4c70679-kube-api-access-qf4sz\") pod \"payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:10.846898 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.846880 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec6f49d-4114-4945-929c-f5a6b4c70679-plugins-config-volume\") pod \"payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:10.847906 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.847862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec6f49d-4114-4945-929c-f5a6b4c70679-plugins-config-volume\") pod \"payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:10.904114 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:10.904081 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4sz\" (UniqueName: \"kubernetes.io/projected/6ec6f49d-4114-4945-929c-f5a6b4c70679-kube-api-access-qf4sz\") pod \"payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:11.018761 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.018183 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:11.113393 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.113329 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6"] Nov 03 21:00:11.118378 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:00:11.118344 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f6d2d1_a249_418c_9b6b_832ad3f7f67d.slice/crio-70760ba977f6a193e3e8918f61cdaf888d023f76f80a1691600a9293fedeee3b WatchSource:0}: Error finding container 70760ba977f6a193e3e8918f61cdaf888d023f76f80a1691600a9293fedeee3b: Status 404 returned error can't find the container with id 70760ba977f6a193e3e8918f61cdaf888d023f76f80a1691600a9293fedeee3b Nov 03 21:00:11.392316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.392234 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6"] Nov 03 21:00:11.398330 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:00:11.398300 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec6f49d_4114_4945_929c_f5a6b4c70679.slice/crio-fdb301cf9d81b5a9328c7635fe2b08509fc308d2e102a18b07afb3cc87b25420 WatchSource:0}: Error finding container fdb301cf9d81b5a9328c7635fe2b08509fc308d2e102a18b07afb3cc87b25420: Status 404 returned error can't find the container with id fdb301cf9d81b5a9328c7635fe2b08509fc308d2e102a18b07afb3cc87b25420 Nov 03 21:00:11.824007 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.823970 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" event={"ID":"6ec6f49d-4114-4945-929c-f5a6b4c70679","Type":"ContainerStarted","Data":"8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1"} Nov 03 21:00:11.824224 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.824035 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" event={"ID":"6ec6f49d-4114-4945-929c-f5a6b4c70679","Type":"ContainerStarted","Data":"fdb301cf9d81b5a9328c7635fe2b08509fc308d2e102a18b07afb3cc87b25420"} Nov 03 21:00:11.824224 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.824154 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:11.825527 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.825503 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" event={"ID":"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d","Type":"ContainerStarted","Data":"4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d"} Nov 03 21:00:11.825649 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.825541 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" event={"ID":"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d","Type":"ContainerStarted","Data":"70760ba977f6a193e3e8918f61cdaf888d023f76f80a1691600a9293fedeee3b"} Nov 03 21:00:11.825649 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.825563 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:11.922179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.916870 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5"] Nov 03 21:00:11.930973 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.929311 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:11.992778 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.992747 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-211d68bf\"/\"openshift-service-ca.crt\"" Nov 03 21:00:11.994441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.992939 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-211d68bf\"/\"default-dockercfg-llmn9\"" Nov 03 21:00:11.994441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.993277 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-211d68bf\"/\"kube-root-ca.crt\"" Nov 03 21:00:11.994441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.994251 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-211d68bf\"/\"e2e-lifecycle-model-211d68bf-kserve-self-signed-certs\"" Nov 03 21:00:11.994441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.994400 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" podStartSLOduration=1.994389112 podStartE2EDuration="1.994389112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:11.949445553 +0000 UTC m=+1879.186423427" watchObservedRunningTime="2026-11-03 21:00:11.994389112 +0000 UTC m=+1879.231367039" Nov 03 21:00:11.996564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:11.996537 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5"] Nov 03 21:00:12.074068 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.073977 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-home\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.074248 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.074118 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e32d66d-1706-4c7f-8465-727dc51387f9-tls-certs\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.074248 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.074176 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-model-cache\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.074383 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.074292 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-dshm\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.074383 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.074358 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-tmp-dir\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.074383 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.074376 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87psz\" (UniqueName: \"kubernetes.io/projected/7e32d66d-1706-4c7f-8465-727dc51387f9-kube-api-access-87psz\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.184397 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-tmp-dir\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.184455 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87psz\" (UniqueName: \"kubernetes.io/projected/7e32d66d-1706-4c7f-8465-727dc51387f9-kube-api-access-87psz\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.184516 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-home\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.184671 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e32d66d-1706-4c7f-8465-727dc51387f9-tls-certs\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.184725 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-model-cache\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.184838 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-dshm\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.186583 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-home\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.186855 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-tmp-dir\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.192624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.188053 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-model-cache\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.201731 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.201703 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e32d66d-1706-4c7f-8465-727dc51387f9-tls-certs\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.211655 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.211479 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-dshm\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.240097 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.240067 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87psz\" (UniqueName: \"kubernetes.io/projected/7e32d66d-1706-4c7f-8465-727dc51387f9-kube-api-access-87psz\") pod \"e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.298358 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.298324 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:12.877846 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.877675 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" podStartSLOduration=2.877625846 podStartE2EDuration="2.877625846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:12.17855725 +0000 UTC m=+1879.415535124" watchObservedRunningTime="2026-11-03 21:00:12.877625846 +0000 UTC m=+1880.114603722" Nov 03 21:00:12.880520 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:12.880496 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5"] Nov 03 21:00:12.886815 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:00:12.886788 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e32d66d_1706_4c7f_8465_727dc51387f9.slice/crio-8c3249a11f458d835cd7f8b74d557fa4cd17382152dcb6dcc42a60c5032fa7aa WatchSource:0}: Error finding container 8c3249a11f458d835cd7f8b74d557fa4cd17382152dcb6dcc42a60c5032fa7aa: Status 404 returned error can't find the container with id 8c3249a11f458d835cd7f8b74d557fa4cd17382152dcb6dcc42a60c5032fa7aa Nov 03 21:00:13.839260 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:13.839217 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" event={"ID":"7e32d66d-1706-4c7f-8465-727dc51387f9","Type":"ContainerStarted","Data":"4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168"} Nov 03 21:00:13.839260 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:13.839264 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" event={"ID":"7e32d66d-1706-4c7f-8465-727dc51387f9","Type":"ContainerStarted","Data":"8c3249a11f458d835cd7f8b74d557fa4cd17382152dcb6dcc42a60c5032fa7aa"} Nov 03 21:00:13.965539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:13.965484 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" podStartSLOduration=2.965468377 podStartE2EDuration="2.965468377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:13.963128135 +0000 UTC m=+1881.200106009" watchObservedRunningTime="2026-11-03 21:00:13.965468377 +0000 UTC m=+1881.202446250" Nov 03 21:00:14.299234 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:14.299193 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:16.912557 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:16.906474 4227 scope.go:122] "RemoveContainer" containerID="7d7a68093ceb1d9061eadec33671c2aa7c886673545e9b9bf5ed8bc55c67c3e9" Nov 03 21:00:17.921035 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:17.921002 4227 scope.go:122] "RemoveContainer" containerID="3a18672eb990e01fb1a35eac6da9a6331407a202ee5dc046feb7d4d62e1cd703" Nov 03 21:00:17.980950 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:17.980911 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" event={"ID":"ab570580-6267-4e2b-a2fe-444e8116a935","Type":"ContainerStarted","Data":"f36406f29048d594de921ee6a8804e24c3c562477236965e123c472e7b4d4e12"} Nov 03 21:00:19.049941 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.049828 4227 generic.go:356] "Generic (PLEG): container finished" podID="ab570580-6267-4e2b-a2fe-444e8116a935" containerID="f36406f29048d594de921ee6a8804e24c3c562477236965e123c472e7b4d4e12" exitCode=60 Nov 03 21:00:19.049941 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.049895 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" event={"ID":"ab570580-6267-4e2b-a2fe-444e8116a935","Type":"ContainerDied","Data":"f36406f29048d594de921ee6a8804e24c3c562477236965e123c472e7b4d4e12"} Nov 03 21:00:19.050564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.049967 4227 scope.go:122] "RemoveContainer" containerID="7d7a68093ceb1d9061eadec33671c2aa7c886673545e9b9bf5ed8bc55c67c3e9" Nov 03 21:00:19.051965 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.051945 4227 scope.go:122] "RemoveContainer" containerID="f36406f29048d594de921ee6a8804e24c3c562477236965e123c472e7b4d4e12" Nov 03 21:00:19.052545 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:19.052520 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78_odh-ai-gateway-infra(ab570580-6267-4e2b-a2fe-444e8116a935)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" Nov 03 21:00:19.052819 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.052795 4227 generic.go:356] "Generic (PLEG): container finished" podID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerID="bd1dee1fb7a337def3ce70ea8101d5b7491e61602638363086c1b3750cdab544" exitCode=60 Nov 03 21:00:19.052922 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.052906 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" event={"ID":"a47a13bb-cdc0-43b3-9766-9cff9d518d40","Type":"ContainerDied","Data":"bd1dee1fb7a337def3ce70ea8101d5b7491e61602638363086c1b3750cdab544"} Nov 03 21:00:19.053349 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.053331 4227 scope.go:122] "RemoveContainer" containerID="bd1dee1fb7a337def3ce70ea8101d5b7491e61602638363086c1b3750cdab544" Nov 03 21:00:19.053696 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:19.053673 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz_odh-ai-gateway-infra(a47a13bb-cdc0-43b3-9766-9cff9d518d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" Nov 03 21:00:19.063409 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.063388 4227 scope.go:122] "RemoveContainer" containerID="3a18672eb990e01fb1a35eac6da9a6331407a202ee5dc046feb7d4d62e1cd703" Nov 03 21:00:19.096827 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:19.096800 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78"] Nov 03 21:00:20.231937 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.231914 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:20.312718 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.312646 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz"] Nov 03 21:00:20.345732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.345698 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab570580-6267-4e2b-a2fe-444e8116a935-kube-api-access-b9nhg\" (UniqueName: \"kubernetes.io/projected/ab570580-6267-4e2b-a2fe-444e8116a935-kube-api-access-b9nhg\") pod \"ab570580-6267-4e2b-a2fe-444e8116a935\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " Nov 03 21:00:20.345919 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.345848 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab570580-6267-4e2b-a2fe-444e8116a935-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab570580-6267-4e2b-a2fe-444e8116a935-maas-api-service-ca\") pod \"ab570580-6267-4e2b-a2fe-444e8116a935\" (UID: \"ab570580-6267-4e2b-a2fe-444e8116a935\") " Nov 03 21:00:20.346892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.346863 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab570580-6267-4e2b-a2fe-444e8116a935-maas-api-service-ca" pod "ab570580-6267-4e2b-a2fe-444e8116a935" (UID: "ab570580-6267-4e2b-a2fe-444e8116a935"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.350705 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.348517 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab570580-6267-4e2b-a2fe-444e8116a935-kube-api-access-b9nhg" pod "ab570580-6267-4e2b-a2fe-444e8116a935" (UID: "ab570580-6267-4e2b-a2fe-444e8116a935"). InnerVolumeSpecName "kube-api-access-b9nhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.447500 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.447471 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9nhg\" (UniqueName: \"kubernetes.io/projected/ab570580-6267-4e2b-a2fe-444e8116a935-kube-api-access-b9nhg\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.447500 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.447497 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab570580-6267-4e2b-a2fe-444e8116a935-maas-api-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.488256 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.488234 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:20.650365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.650336 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47a13bb-cdc0-43b3-9766-9cff9d518d40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a13bb-cdc0-43b3-9766-9cff9d518d40-maas-api-service-ca\") pod \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " Nov 03 21:00:20.650565 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.650551 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47a13bb-cdc0-43b3-9766-9cff9d518d40-kube-api-access-qfd7w\" (UniqueName: \"kubernetes.io/projected/a47a13bb-cdc0-43b3-9766-9cff9d518d40-kube-api-access-qfd7w\") pod \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\" (UID: \"a47a13bb-cdc0-43b3-9766-9cff9d518d40\") " Nov 03 21:00:20.650707 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.650682 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47a13bb-cdc0-43b3-9766-9cff9d518d40-maas-api-service-ca" pod "a47a13bb-cdc0-43b3-9766-9cff9d518d40" (UID: "a47a13bb-cdc0-43b3-9766-9cff9d518d40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.650939 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.650921 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47a13bb-cdc0-43b3-9766-9cff9d518d40-maas-api-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.652652 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.652633 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47a13bb-cdc0-43b3-9766-9cff9d518d40-kube-api-access-qfd7w" pod "a47a13bb-cdc0-43b3-9766-9cff9d518d40" (UID: "a47a13bb-cdc0-43b3-9766-9cff9d518d40"). InnerVolumeSpecName "kube-api-access-qfd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.751970 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:20.751937 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfd7w\" (UniqueName: \"kubernetes.io/projected/a47a13bb-cdc0-43b3-9766-9cff9d518d40-kube-api-access-qfd7w\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.071826 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.071789 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" event={"ID":"ab570580-6267-4e2b-a2fe-444e8116a935","Type":"ContainerDied","Data":"b7d2bce228a4be8cbe29ff4a63fb90742fba6b8cddbd7df111d06e4f1c29b7c1"} Nov 03 21:00:21.071826 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.071792 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78" Nov 03 21:00:21.072090 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.071905 4227 scope.go:122] "RemoveContainer" containerID="f36406f29048d594de921ee6a8804e24c3c562477236965e123c472e7b4d4e12" Nov 03 21:00:21.074495 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.074453 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" event={"ID":"a47a13bb-cdc0-43b3-9766-9cff9d518d40","Type":"ContainerDied","Data":"91a1874880e70280f26f988b0371c61361d8374d584108799604b2f5fb290bb6"} Nov 03 21:00:21.074626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.074538 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz" Nov 03 21:00:21.084085 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.084067 4227 scope.go:122] "RemoveContainer" containerID="bd1dee1fb7a337def3ce70ea8101d5b7491e61602638363086c1b3750cdab544" Nov 03 21:00:21.225333 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.225305 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz"] Nov 03 21:00:21.233960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.233935 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895660-dqjmz"] Nov 03 21:00:21.332182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.332143 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78"] Nov 03 21:00:21.336280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:21.336257 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895660-s5b78"] Nov 03 21:00:22.298970 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:22.298941 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:22.311764 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:22.311736 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:22.835628 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:22.835595 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:00:22.835871 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:22.835854 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:00:22.901393 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:22.901365 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" path="/var/lib/kubelet/pods/a47a13bb-cdc0-43b3-9766-9cff9d518d40/volumes" Nov 03 21:00:22.901786 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:22.901772 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" path="/var/lib/kubelet/pods/ab570580-6267-4e2b-a2fe-444e8116a935/volumes" Nov 03 21:00:23.106610 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:23.106534 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:00:26.086074 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:26.085980 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:00:42.785795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.785761 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght"] Nov 03 21:00:42.786344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786327 4227 cpu_manager.go:414] "containerMap: removing container" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786345 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786357 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786362 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786368 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786374 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786393 4227 cpu_manager.go:414] "containerMap: removing container" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786399 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786406 4227 cpu_manager.go:414] "containerMap: removing container" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786410 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786506 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786513 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:42.786653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786524 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:42.786653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786529 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:42.786653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.786539 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:42.791305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.791282 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814533 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-istio-token\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814601 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66991d08-963b-470c-836c-ee803c475c1f-istio-podinfo\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814668 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-certs\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814772 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-credential-socket\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814826 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66991d08-963b-470c-836c-ee803c475c1f-istiod-ca-cert\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814854 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-socket\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814892 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-data\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814936 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-envoy\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:42.817180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:42.814985 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgtc\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-kube-api-access-ftgtc\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071043 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-istio-token\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071128 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66991d08-963b-470c-836c-ee803c475c1f-istio-podinfo\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071213 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-certs\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071316 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-credential-socket\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071375 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66991d08-963b-470c-836c-ee803c475c1f-istiod-ca-cert\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071401 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-socket\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071437 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-data\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071485 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-envoy\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.073773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.071534 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgtc\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-kube-api-access-ftgtc\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.123218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.116915 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-certs\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.123218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.117190 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-credential-socket\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.123218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.119384 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e6aaa6-openshift-default-dockercfg-v6x8p\"" Nov 03 21:00:43.123218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.119680 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-data\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.123218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.120007 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66991d08-963b-470c-836c-ee803c475c1f-istiod-ca-cert\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.137190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.130638 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-socket\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.197451 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.196940 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-istio-token\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.241296 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.241256 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght"] Nov 03 21:00:43.248061 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.242702 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgtc\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-kube-api-access-ftgtc\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.283544 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.283094 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-envoy\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.283544 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.283440 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66991d08-963b-470c-836c-ee803c475c1f-istio-podinfo\") pod \"e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.410182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.407383 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:43.860836 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.860743 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght"] Nov 03 21:00:43.867613 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:00:43.867588 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66991d08_963b_470c_836c_ee803c475c1f.slice/crio-1d63a4b6a4b32e409f6d9552c302a32da73ed3265465257b244d222c140706e3 WatchSource:0}: Error finding container 1d63a4b6a4b32e409f6d9552c302a32da73ed3265465257b244d222c140706e3: Status 404 returned error can't find the container with id 1d63a4b6a4b32e409f6d9552c302a32da73ed3265465257b244d222c140706e3 Nov 03 21:00:43.870769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.870733 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:43.870876 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.870814 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:43.870876 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:43.870865 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:44.254740 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:44.254701 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" event={"ID":"66991d08-963b-470c-836c-ee803c475c1f","Type":"ContainerStarted","Data":"449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c"} Nov 03 21:00:44.254937 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:44.254766 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" event={"ID":"66991d08-963b-470c-836c-ee803c475c1f","Type":"ContainerStarted","Data":"1d63a4b6a4b32e409f6d9552c302a32da73ed3265465257b244d222c140706e3"} Nov 03 21:00:44.369767 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:44.369715 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" podStartSLOduration=2.3697022 podStartE2EDuration="2.3697022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:44.366433054 +0000 UTC m=+1911.603410928" watchObservedRunningTime="2026-11-03 21:00:44.3697022 +0000 UTC m=+1911.606680073" Nov 03 21:00:44.408273 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:44.408235 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:44.414848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:44.414821 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:45.262078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:45.262044 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:45.263031 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:45.263012 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:00:50.995191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:50.989641 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464"] Nov 03 21:00:50.995191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:50.992928 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:50.995191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:50.992951 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab570580-6267-4e2b-a2fe-444e8116a935" containerName="cleanup" Nov 03 21:00:50.995191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:50.993127 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47a13bb-cdc0-43b3-9766-9cff9d518d40" containerName="cleanup" Nov 03 21:00:51.001876 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.001850 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464"] Nov 03 21:00:51.002191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.002102 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.095187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.094381 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ss2\" (UniqueName: \"kubernetes.io/projected/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-kube-api-access-w7ss2\") pod \"payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.095187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.094449 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.095468 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.095318 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e6aaa6\"" Nov 03 21:00:51.097508 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.095581 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e6aaa6-dockercfg-lr5q7\"" Nov 03 21:00:51.234414 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.198289 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ss2\" (UniqueName: \"kubernetes.io/projected/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-kube-api-access-w7ss2\") pod \"payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.234718 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.234689 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.260287 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.236671 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.260287 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.237337 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4"] Nov 03 21:00:51.260287 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.259450 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4"] Nov 03 21:00:51.260287 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.259567 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.433486 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.433049 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ss2\" (UniqueName: \"kubernetes.io/projected/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-kube-api-access-w7ss2\") pod \"payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.453425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.453390 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:51.487518 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.487476 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9eb92-19eb-436a-a7d7-515d9e22af7c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.487670 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.487600 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4f4\" (UniqueName: \"kubernetes.io/projected/84d9eb92-19eb-436a-a7d7-515d9e22af7c-kube-api-access-5n4f4\") pod \"payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.588911 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.588873 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9eb92-19eb-436a-a7d7-515d9e22af7c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.589172 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.588974 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4f4\" (UniqueName: \"kubernetes.io/projected/84d9eb92-19eb-436a-a7d7-515d9e22af7c-kube-api-access-5n4f4\") pod \"payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.590274 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.590244 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9eb92-19eb-436a-a7d7-515d9e22af7c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.642325 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.642288 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4f4\" (UniqueName: \"kubernetes.io/projected/84d9eb92-19eb-436a-a7d7-515d9e22af7c-kube-api-access-5n4f4\") pod \"payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.747425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.747399 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:51.838604 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:51.838576 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464"] Nov 03 21:00:51.844309 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:00:51.844283 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e970a1f_cb48_4d6c_aa8e_7ff6354f3d1d.slice/crio-e3471268832ae212a503dfc2c2d68b33d11da418e8e716fef51a05e97ab643a8 WatchSource:0}: Error finding container e3471268832ae212a503dfc2c2d68b33d11da418e8e716fef51a05e97ab643a8: Status 404 returned error can't find the container with id e3471268832ae212a503dfc2c2d68b33d11da418e8e716fef51a05e97ab643a8 Nov 03 21:00:52.131272 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.131187 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4"] Nov 03 21:00:52.135891 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:00:52.135857 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d9eb92_19eb_436a_a7d7_515d9e22af7c.slice/crio-414bc0c7f3cf42901d2e4d8605c2e9cd4c73dfc9dbfa3b60083301748c26217b WatchSource:0}: Error finding container 414bc0c7f3cf42901d2e4d8605c2e9cd4c73dfc9dbfa3b60083301748c26217b: Status 404 returned error can't find the container with id 414bc0c7f3cf42901d2e4d8605c2e9cd4c73dfc9dbfa3b60083301748c26217b Nov 03 21:00:52.447452 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.447411 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" event={"ID":"84d9eb92-19eb-436a-a7d7-515d9e22af7c","Type":"ContainerStarted","Data":"082a76294b04780257d1bcef9aac20a29dd9fab087d08ed9f50bd744a7e77a9d"} Nov 03 21:00:52.447626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.447517 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:00:52.447626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.447544 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" event={"ID":"84d9eb92-19eb-436a-a7d7-515d9e22af7c","Type":"ContainerStarted","Data":"414bc0c7f3cf42901d2e4d8605c2e9cd4c73dfc9dbfa3b60083301748c26217b"} Nov 03 21:00:52.449311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.449276 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" event={"ID":"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d","Type":"ContainerStarted","Data":"41f034dae5f66e4cd0c8fd07fb76f02baeb3551bcb1df9c2789c39f77b5fd961"} Nov 03 21:00:52.449417 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.449336 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" event={"ID":"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d","Type":"ContainerStarted","Data":"e3471268832ae212a503dfc2c2d68b33d11da418e8e716fef51a05e97ab643a8"} Nov 03 21:00:52.449417 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.449363 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:00:52.554445 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.554391 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" podStartSLOduration=1.554377165 podStartE2EDuration="1.554377165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:52.553392066 +0000 UTC m=+1919.790369941" watchObservedRunningTime="2026-11-03 21:00:52.554377165 +0000 UTC m=+1919.791355038" Nov 03 21:00:52.662570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.662510 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" podStartSLOduration=2.662492081 podStartE2EDuration="2.662492081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:52.658853628 +0000 UTC m=+1919.895831503" watchObservedRunningTime="2026-11-03 21:00:52.662492081 +0000 UTC m=+1919.899469960" Nov 03 21:00:52.820005 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.819930 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg"] Nov 03 21:00:52.823738 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.823720 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921517 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921589 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921729 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08577ba9-52bb-4c8c-8879-63deb9968296-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921768 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921794 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921837 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921892 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921930 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2ck\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-kube-api-access-sb2ck\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.921973 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08577ba9-52bb-4c8c-8879-63deb9968296-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:52.928369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.922455 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-20d356-openshift-default-dockercfg-fndvt\"" Nov 03 21:00:52.991881 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:52.991845 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg"] Nov 03 21:00:53.023091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023060 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023222 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08577ba9-52bb-4c8c-8879-63deb9968296-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023276 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023300 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023489 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023345 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023489 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023421 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023489 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023446 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023489 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023460 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2ck\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-kube-api-access-sb2ck\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023688 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023507 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08577ba9-52bb-4c8c-8879-63deb9968296-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023688 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023604 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023688 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023657 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.023980 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.023956 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.024592 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.024568 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.024835 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.024800 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08577ba9-52bb-4c8c-8879-63deb9968296-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.026786 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.026760 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.027220 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.027203 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08577ba9-52bb-4c8c-8879-63deb9968296-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.078042 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.077901 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.079264 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.079240 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2ck\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-kube-api-access-sb2ck\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.142427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.142393 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:53.839507 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.839437 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg"] Nov 03 21:00:53.846440 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:00:53.846411 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08577ba9_52bb_4c8c_8879_63deb9968296.slice/crio-788d3949bcf5184dd935efd1b72b70774d237207c53efdea753b90abaa9061bd WatchSource:0}: Error finding container 788d3949bcf5184dd935efd1b72b70774d237207c53efdea753b90abaa9061bd: Status 404 returned error can't find the container with id 788d3949bcf5184dd935efd1b72b70774d237207c53efdea753b90abaa9061bd Nov 03 21:00:53.849078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.849044 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:53.849206 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.849120 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:53.849206 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:53.849150 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:00:54.484339 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:54.484295 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" event={"ID":"08577ba9-52bb-4c8c-8879-63deb9968296","Type":"ContainerStarted","Data":"7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53"} Nov 03 21:00:54.484867 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:54.484391 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" event={"ID":"08577ba9-52bb-4c8c-8879-63deb9968296","Type":"ContainerStarted","Data":"788d3949bcf5184dd935efd1b72b70774d237207c53efdea753b90abaa9061bd"} Nov 03 21:00:54.598343 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:54.598262 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" podStartSLOduration=2.598243197 podStartE2EDuration="2.598243197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:54.592881809 +0000 UTC m=+1921.829859695" watchObservedRunningTime="2026-11-03 21:00:54.598243197 +0000 UTC m=+1921.835221073" Nov 03 21:00:55.143110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:55.143074 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:55.149264 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:55.149236 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:55.490517 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:55.490478 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:55.491333 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:00:55.491313 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:00:56.081533 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:00:56.081496 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:01:03.462746 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:03.462716 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:01:03.463271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:03.462786 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:01:10.344532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.344317 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn"] Nov 03 21:01:10.363182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.361929 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn"] Nov 03 21:01:10.363182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.362047 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:10.458101 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.458067 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-20d356\"" Nov 03 21:01:10.458291 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.458136 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-20d356-dockercfg-cpw55\"" Nov 03 21:01:10.512429 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.512393 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t88p6\" (UniqueName: \"kubernetes.io/projected/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-kube-api-access-t88p6\") pod \"payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:10.512596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.512516 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:10.613678 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.613602 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88p6\" (UniqueName: \"kubernetes.io/projected/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-kube-api-access-t88p6\") pod \"payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:10.613818 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.613689 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:10.614415 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.614397 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:10.666515 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.666491 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t88p6\" (UniqueName: \"kubernetes.io/projected/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-kube-api-access-t88p6\") pod \"payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:10.686307 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:10.686286 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:11.058232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:11.058199 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn"] Nov 03 21:01:11.064660 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:01:11.064625 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41292328_b7e4_4c4e_8bd9_e17ce0f867dc.slice/crio-244c9a5efb50c077b23b07ff8702b8f80613cdedfb4847e67e58f5c905b8cd46 WatchSource:0}: Error finding container 244c9a5efb50c077b23b07ff8702b8f80613cdedfb4847e67e58f5c905b8cd46: Status 404 returned error can't find the container with id 244c9a5efb50c077b23b07ff8702b8f80613cdedfb4847e67e58f5c905b8cd46 Nov 03 21:01:11.636784 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:11.636746 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" event={"ID":"41292328-b7e4-4c4e-8bd9-e17ce0f867dc","Type":"ContainerStarted","Data":"36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c"} Nov 03 21:01:11.637208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:11.636799 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" event={"ID":"41292328-b7e4-4c4e-8bd9-e17ce0f867dc","Type":"ContainerStarted","Data":"244c9a5efb50c077b23b07ff8702b8f80613cdedfb4847e67e58f5c905b8cd46"} Nov 03 21:01:11.637208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:11.636842 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:11.744950 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:11.744900 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" podStartSLOduration=1.744886489 podStartE2EDuration="1.744886489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:11.742236732 +0000 UTC m=+1938.979214607" watchObservedRunningTime="2026-11-03 21:01:11.744886489 +0000 UTC m=+1938.981864362" Nov 03 21:01:13.775531 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.775480 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx"] Nov 03 21:01:13.780621 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.780597 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.831871 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.831838 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx"] Nov 03 21:01:13.865551 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.865508 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtcq\" (UniqueName: \"kubernetes.io/projected/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-kube-api-access-2vtcq\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.865769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.865614 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-metrics-certs\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.865769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.865702 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.865913 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.865858 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-openshift-service-ca\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.873495 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.873465 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-46ca0b-serving-cert\"" Nov 03 21:01:13.967471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.967430 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtcq\" (UniqueName: \"kubernetes.io/projected/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-kube-api-access-2vtcq\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.967679 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.967530 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-metrics-certs\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.967679 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.967620 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.967806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.967770 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-openshift-service-ca\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.968691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.968660 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-openshift-service-ca\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:13.969438 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:01:13.969413 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-serving-cert: secret "maas-api-e2e-api-a-46ca0b-serving-cert" not found Nov 03 21:01:13.969602 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:01:13.969484 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls podName:9e50bbbc-e6d8-4381-b4e2-bee22edb8620 nodeName:}" failed. No retries permitted until 2026-11-03 21:01:14.469464972 +0000 UTC m=+1941.706442839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls") pod "maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" (UID: "9e50bbbc-e6d8-4381-b4e2-bee22edb8620") : secret "maas-api-e2e-api-a-46ca0b-serving-cert" not found Nov 03 21:01:13.975366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:13.975336 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-metrics-certs\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:14.037009 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:14.036931 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtcq\" (UniqueName: \"kubernetes.io/projected/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-kube-api-access-2vtcq\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:14.482093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:14.482060 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:14.484873 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:14.484840 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls\") pod \"maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:14.696756 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:14.696721 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:15.075351 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:15.075270 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx"] Nov 03 21:01:15.081246 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:01:15.081213 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e50bbbc_e6d8_4381_b4e2_bee22edb8620.slice/crio-3489655aef9368506d9d9a65679f6e2fd96f23cd0aa2ddc71d5371d058717814 WatchSource:0}: Error finding container 3489655aef9368506d9d9a65679f6e2fd96f23cd0aa2ddc71d5371d058717814: Status 404 returned error can't find the container with id 3489655aef9368506d9d9a65679f6e2fd96f23cd0aa2ddc71d5371d058717814 Nov 03 21:01:15.666082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:15.666046 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" event={"ID":"9e50bbbc-e6d8-4381-b4e2-bee22edb8620","Type":"ContainerStarted","Data":"3489655aef9368506d9d9a65679f6e2fd96f23cd0aa2ddc71d5371d058717814"} Nov 03 21:01:16.674381 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:16.674345 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" event={"ID":"9e50bbbc-e6d8-4381-b4e2-bee22edb8620","Type":"ContainerStarted","Data":"e7f5d8af99467fb5cda4c769d9b406a86e304ebf20a3a986696483ddb73959c5"} Nov 03 21:01:16.674797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:16.674448 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:22.648030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:22.648000 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:01:22.686440 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:22.686412 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:01:22.756318 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:01:22.756260 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" podStartSLOduration=8.871065884 podStartE2EDuration="9.756244267s" totalImagesPullingTime="885.178383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:16.785402384 +0000 UTC m=+1944.022380262" watchObservedRunningTime="2026-11-03 21:01:22.756244267 +0000 UTC m=+1949.993222205" Nov 03 21:01:26.082308 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:01:26.082265 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:01:56.082546 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:01:56.082512 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:02:11.033185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.019241 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch"] Nov 03 21:02:11.060182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.054578 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.096184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.076994 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch"] Nov 03 21:02:11.127183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.126249 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.127183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.126299 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.127183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.126519 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4f7l\" (UniqueName: \"kubernetes.io/projected/c2507a90-02cd-4107-9df9-6fba6a9240bd-kube-api-access-j4f7l\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.127183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.126636 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2507a90-02cd-4107-9df9-6fba6a9240bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.147180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.127991 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-c63dc3-serving-cert\"" Nov 03 21:02:11.253023 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.252986 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2507a90-02cd-4107-9df9-6fba6a9240bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.253374 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.253347 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.253519 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.253503 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.253834 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.253812 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f7l\" (UniqueName: \"kubernetes.io/projected/c2507a90-02cd-4107-9df9-6fba6a9240bd-kube-api-access-j4f7l\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.255868 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.255847 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2507a90-02cd-4107-9df9-6fba6a9240bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.267138 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:02:11.267117 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-serving-cert: secret "maas-api-e2e-shared-b-w6-c63dc3-serving-cert" not found Nov 03 21:02:11.267425 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:02:11.267349 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls podName:c2507a90-02cd-4107-9df9-6fba6a9240bd nodeName:}" failed. No retries permitted until 2026-11-03 21:02:11.767328697 +0000 UTC m=+1999.004306554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls") pod "maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" (UID: "c2507a90-02cd-4107-9df9-6fba6a9240bd") : secret "maas-api-e2e-shared-b-w6-c63dc3-serving-cert" not found Nov 03 21:02:11.287246 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.278035 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.410875 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.410845 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f7l\" (UniqueName: \"kubernetes.io/projected/c2507a90-02cd-4107-9df9-6fba6a9240bd-kube-api-access-j4f7l\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.811664 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.811627 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.814007 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.813978 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:11.879445 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:11.879403 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:12.394266 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:12.394232 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch"] Nov 03 21:02:12.400093 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:02:12.400052 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2507a90_02cd_4107_9df9_6fba6a9240bd.slice/crio-f0dae2aca6da0c8a48e504388c6f4494a0dedb3f022d0478cd8d44f2b493070f WatchSource:0}: Error finding container f0dae2aca6da0c8a48e504388c6f4494a0dedb3f022d0478cd8d44f2b493070f: Status 404 returned error can't find the container with id f0dae2aca6da0c8a48e504388c6f4494a0dedb3f022d0478cd8d44f2b493070f Nov 03 21:02:13.300020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:13.299980 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" event={"ID":"c2507a90-02cd-4107-9df9-6fba6a9240bd","Type":"ContainerStarted","Data":"f0dae2aca6da0c8a48e504388c6f4494a0dedb3f022d0478cd8d44f2b493070f"} Nov 03 21:02:14.311689 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:14.311656 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" event={"ID":"c2507a90-02cd-4107-9df9-6fba6a9240bd","Type":"ContainerStarted","Data":"80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255"} Nov 03 21:02:14.312192 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:14.311927 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:14.419821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:14.419773 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" podStartSLOduration=3.491082475 podStartE2EDuration="4.419759815s" totalImagesPullingTime="928.67734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:14.41774188 +0000 UTC m=+2001.654719751" watchObservedRunningTime="2026-11-03 21:02:14.419759815 +0000 UTC m=+2001.656737689" Nov 03 21:02:20.325660 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:20.325629 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:02:25.478365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.478330 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt"] Nov 03 21:02:25.482824 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.482803 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.536352 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.536322 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt"] Nov 03 21:02:25.577453 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.577420 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-20d356-serving-cert\"" Nov 03 21:02:25.679118 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.679087 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.679289 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.679261 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.679399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.679385 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.679458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.679447 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtw8\" (UniqueName: \"kubernetes.io/projected/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-kube-api-access-zqtw8\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.783247 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.783138 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtw8\" (UniqueName: \"kubernetes.io/projected/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-kube-api-access-zqtw8\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.783568 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.783543 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.783819 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.783801 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.784036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.784020 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.801043 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.801007 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.807907 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.807849 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.808798 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.808758 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:25.837596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:25.837563 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtw8\" (UniqueName: \"kubernetes.io/projected/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-kube-api-access-zqtw8\") pod \"maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:26.082249 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:02:26.082139 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:02:26.096771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:26.096746 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:26.500596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:26.498302 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt"] Nov 03 21:02:27.417125 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:27.417083 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" event={"ID":"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3","Type":"ContainerStarted","Data":"6139dffa865b9f63bd950cbe3e364253896662d5b17b7cbf2536664ad2c3e052"} Nov 03 21:02:28.425456 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:28.425421 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" event={"ID":"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3","Type":"ContainerStarted","Data":"cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb"} Nov 03 21:02:28.425953 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:28.425533 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:28.564182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:28.558388 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" podStartSLOduration=2.6248404340000002 podStartE2EDuration="3.558364973s" totalImagesPullingTime="933.524539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:28.53366348 +0000 UTC m=+2015.770641356" watchObservedRunningTime="2026-11-03 21:02:28.558364973 +0000 UTC m=+2015.795342848" Nov 03 21:02:34.439228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:34.439142 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:02:50.606442 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:50.606362 4227 generic.go:356] "Generic (PLEG): container finished" podID="66991d08-963b-470c-836c-ee803c475c1f" containerID="449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c" exitCode=0 Nov 03 21:02:50.606875 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:50.606431 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" event={"ID":"66991d08-963b-470c-836c-ee803c475c1f","Type":"ContainerDied","Data":"449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c"} Nov 03 21:02:50.606875 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:50.606849 4227 scope.go:122] "RemoveContainer" containerID="449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c" Nov 03 21:02:50.608584 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:50.608559 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:02:50.608646 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:50.608627 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:02:50.608684 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:50.608662 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:02:51.614681 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:51.614646 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" event={"ID":"66991d08-963b-470c-836c-ee803c475c1f","Type":"ContainerStarted","Data":"5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74"} Nov 03 21:02:52.408673 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:52.408638 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:02:52.408893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:52.408870 4227 patch_prober.go:29] interesting pod/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 21:02:52.408941 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:52.408921 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 21:02:53.409034 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:53.408999 4227 patch_prober.go:29] interesting pod/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 21:02:53.409462 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:53.409054 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 21:02:54.408840 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:54.408800 4227 patch_prober.go:29] interesting pod/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 21:02:54.409016 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:54.408859 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 21:02:55.408028 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:55.407991 4227 patch_prober.go:29] interesting pod/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 21:02:55.408449 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:55.408041 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 21:02:56.108851 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:02:56.108801 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:02:56.408863 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:56.408833 4227 patch_prober.go:29] interesting pod/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 21:02:56.409260 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:56.408889 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 21:02:57.414424 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:57.414390 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:02:57.414836 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:57.414813 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:02:57.415607 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:57.415591 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:02:59.763187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:59.762344 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb"] Nov 03 21:02:59.763187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:02:59.762609 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" containerID="cri-o://c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d" gracePeriod=30 Nov 03 21:03:01.911797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:01.911762 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48"] Nov 03 21:03:01.912291 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:01.912008 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" containerID="cri-o://24cd5f0f735798be57f3250ba476b76fafb58c3c231fe1f6a9429a1e071acf19" gracePeriod=30 Nov 03 21:03:03.196427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:03.196391 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:03:03.196853 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:03.196461 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:03:04.868836 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:04.868802 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:03:04.869254 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:04.868860 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:03:05.156453 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.156422 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 21:03:05.182580 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.182548 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-certs\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.182759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.182606 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35251231-f967-4184-970b-bcb4bf41a0f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35251231-f967-4184-970b-bcb4bf41a0f5-istiod-ca-cert\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.182817 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.182776 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-credential-socket\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.182869 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.182851 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-certs" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:05.182869 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.182859 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35251231-f967-4184-970b-bcb4bf41a0f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35251231-f967-4184-970b-bcb4bf41a0f5-istio-podinfo\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.182979 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.182901 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-istio-token\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-istio-token\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.183042 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.182977 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-kube-api-access-pptbd\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-kube-api-access-pptbd\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.183042 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.183001 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-socket\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.183144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.183057 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-data\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.183144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.183058 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-credential-socket" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:05.183144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.183077 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35251231-f967-4184-970b-bcb4bf41a0f5-istiod-ca-cert" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:05.183144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.183077 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-envoy\") pod \"35251231-f967-4184-970b-bcb4bf41a0f5\" (UID: \"35251231-f967-4184-970b-bcb4bf41a0f5\") " Nov 03 21:03:05.183395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.183372 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-socket" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:05.183616 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.183578 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-data" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:05.184083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.184063 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.184218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.184088 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.184218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.184103 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.184218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.184120 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.184218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.184134 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35251231-f967-4184-970b-bcb4bf41a0f5-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.185409 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.185388 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-envoy" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:05.185638 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.185607 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-kube-api-access-pptbd" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "kube-api-access-pptbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:05.185802 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.185783 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35251231-f967-4184-970b-bcb4bf41a0f5-istio-podinfo" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:05.185900 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.185882 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-istio-token" pod "35251231-f967-4184-970b-bcb4bf41a0f5" (UID: "35251231-f967-4184-970b-bcb4bf41a0f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:05.285434 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.285400 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35251231-f967-4184-970b-bcb4bf41a0f5-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.285434 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.285430 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.285434 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.285440 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptbd\" (UniqueName: \"kubernetes.io/projected/35251231-f967-4184-970b-bcb4bf41a0f5-kube-api-access-pptbd\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.285684 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.285451 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35251231-f967-4184-970b-bcb4bf41a0f5-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:05.524732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.524694 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl"] Nov 03 21:03:05.525318 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.525303 4227 cpu_manager.go:414] "containerMap: removing container" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" Nov 03 21:03:05.525371 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.525319 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" Nov 03 21:03:05.525438 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.525427 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" containerName="istio-proxy" Nov 03 21:03:05.528349 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.528332 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.589181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.588289 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3ca125-openshift-default-dockercfg-rc7wm\"" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591016 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-certs\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591099 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istiod-ca-cert\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591150 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-socket\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591198 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-credential-socket\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591354 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-envoy\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591421 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-podinfo\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591453 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-token\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591540 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bqq\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-kube-api-access-49bqq\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.595178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.591601 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-data\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.622300 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.615318 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl"] Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.849106 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-envoy\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.859729 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-podinfo\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.859781 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-token\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.859897 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bqq\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-kube-api-access-49bqq\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.859979 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-data\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.860065 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-certs\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.860131 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istiod-ca-cert\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.860214 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-socket\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.860326 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-credential-socket\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.861078 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-credential-socket\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.862211 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-data\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.881769 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istiod-ca-cert\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.882073 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-certs\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:05.892250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:05.883258 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-socket\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:06.173183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.170962 4227 generic.go:356] "Generic (PLEG): container finished" podID="35251231-f967-4184-970b-bcb4bf41a0f5" containerID="c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d" exitCode=0 Nov 03 21:03:06.173183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.171089 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" event={"ID":"35251231-f967-4184-970b-bcb4bf41a0f5","Type":"ContainerDied","Data":"c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d"} Nov 03 21:03:06.173183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.171230 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" event={"ID":"35251231-f967-4184-970b-bcb4bf41a0f5","Type":"ContainerDied","Data":"c763ba7304c1ddddcdf5724043fb24828d7c9c62ef8af3f01f97a26a7fa439b1"} Nov 03 21:03:06.173183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.171295 4227 scope.go:122] "RemoveContainer" containerID="c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d" Nov 03 21:03:06.173183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.172398 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb" Nov 03 21:03:06.239195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.237063 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bqq\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-kube-api-access-49bqq\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:06.255182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.244331 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-podinfo\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:06.255182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.244636 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-envoy\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:06.307564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.298919 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-token\") pod \"e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:06.337413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.335822 4227 scope.go:122] "RemoveContainer" containerID="c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d" Nov 03 21:03:06.371202 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:03:06.339627 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d\": container with ID starting with c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d not found: ID does not exist" containerID="c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d" Nov 03 21:03:06.371202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.339682 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d"} err="failed to get container status \"c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d\": rpc error: code = NotFound desc = could not find container \"c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d\": container with ID starting with c6fb7af903c5d1bc754b6115098c470503af9173b59e6cd6151ad541bc04849d not found: ID does not exist" Nov 03 21:03:06.474374 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.473799 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:06.557235 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.555932 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb"] Nov 03 21:03:06.565617 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.565574 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-4fknb"] Nov 03 21:03:06.905132 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:06.905093 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35251231-f967-4184-970b-bcb4bf41a0f5" path="/var/lib/kubelet/pods/35251231-f967-4184-970b-bcb4bf41a0f5/volumes" Nov 03 21:03:07.180754 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.180726 4227 generic.go:356] "Generic (PLEG): container finished" podID="6b082e35-1549-4590-93f3-2713bfdda109" containerID="24cd5f0f735798be57f3250ba476b76fafb58c3c231fe1f6a9429a1e071acf19" exitCode=0 Nov 03 21:03:07.180891 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.180797 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" event={"ID":"6b082e35-1549-4590-93f3-2713bfdda109","Type":"ContainerDied","Data":"24cd5f0f735798be57f3250ba476b76fafb58c3c231fe1f6a9429a1e071acf19"} Nov 03 21:03:07.180891 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.180842 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" event={"ID":"6b082e35-1549-4590-93f3-2713bfdda109","Type":"ContainerDied","Data":"d7d1fdc629481d4608877613c944d9bdf7ab1ac3d2a64e0dbb897a20186a8cee"} Nov 03 21:03:07.180891 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.180858 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d1fdc629481d4608877613c944d9bdf7ab1ac3d2a64e0dbb897a20186a8cee" Nov 03 21:03:07.189335 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.189307 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 21:03:07.238198 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.237760 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl"] Nov 03 21:03:07.244754 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:03:07.242775 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fdb9d7_abfd_4c25_9f60_503543c9a0a7.slice/crio-b6ed545a23758b2b4a3b193a723013f514156bbf064b2c480bb9a5b5db18c9fe WatchSource:0}: Error finding container b6ed545a23758b2b4a3b193a723013f514156bbf064b2c480bb9a5b5db18c9fe: Status 404 returned error can't find the container with id b6ed545a23758b2b4a3b193a723013f514156bbf064b2c480bb9a5b5db18c9fe Nov 03 21:03:07.246294 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.246255 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:07.246412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.246344 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:07.246412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.246389 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:07.261835 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.261745 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-credential-socket\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.261898 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.261856 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b082e35-1549-4590-93f3-2713bfdda109-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b082e35-1549-4590-93f3-2713bfdda109-istio-podinfo\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.261986 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.261968 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b082e35-1549-4590-93f3-2713bfdda109-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b082e35-1549-4590-93f3-2713bfdda109-istiod-ca-cert\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.262050 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.262004 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-credential-socket" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:07.262381 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.262355 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b082e35-1549-4590-93f3-2713bfdda109-istiod-ca-cert" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:07.263199 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263176 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-certs\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.263304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263227 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-socket\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.263304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263286 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-istio-token\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-istio-token\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.263409 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263353 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-envoy\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.263409 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263400 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-kube-api-access-kgsxd\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-kube-api-access-kgsxd\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.263514 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263470 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-certs" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:07.263573 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263509 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-socket" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:07.263637 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.263580 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-data\") pod \"6b082e35-1549-4590-93f3-2713bfdda109\" (UID: \"6b082e35-1549-4590-93f3-2713bfdda109\") " Nov 03 21:03:07.264151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.264131 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b082e35-1549-4590-93f3-2713bfdda109-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.264330 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.264154 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.264330 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.264202 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.264330 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.264217 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.264522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.264341 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-data" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:07.265567 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.265542 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-envoy" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:07.265886 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.265850 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b082e35-1549-4590-93f3-2713bfdda109-istio-podinfo" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:07.265967 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.265909 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-istio-token" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:07.269298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.266422 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-kube-api-access-kgsxd" pod "6b082e35-1549-4590-93f3-2713bfdda109" (UID: "6b082e35-1549-4590-93f3-2713bfdda109"). InnerVolumeSpecName "kube-api-access-kgsxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:07.365729 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.365691 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.365729 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.365723 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.365729 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.365734 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgsxd\" (UniqueName: \"kubernetes.io/projected/6b082e35-1549-4590-93f3-2713bfdda109-kube-api-access-kgsxd\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.366012 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.365743 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b082e35-1549-4590-93f3-2713bfdda109-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:07.366012 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:07.365752 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b082e35-1549-4590-93f3-2713bfdda109-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:08.188337 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.188300 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48" Nov 03 21:03:08.188337 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.188315 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" event={"ID":"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7","Type":"ContainerStarted","Data":"7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5"} Nov 03 21:03:08.188857 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.188362 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" event={"ID":"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7","Type":"ContainerStarted","Data":"b6ed545a23758b2b4a3b193a723013f514156bbf064b2c480bb9a5b5db18c9fe"} Nov 03 21:03:08.393531 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.393413 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" podStartSLOduration=3.393394748 podStartE2EDuration="3.393394748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:08.392718845 +0000 UTC m=+2055.629696718" watchObservedRunningTime="2026-11-03 21:03:08.393394748 +0000 UTC m=+2055.630372624" Nov 03 21:03:08.474936 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.474903 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:08.480532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.480506 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:08.499387 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.499362 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48"] Nov 03 21:03:08.505438 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.505348 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-hbd48"] Nov 03 21:03:08.966241 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:08.966198 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b082e35-1549-4590-93f3-2713bfdda109" path="/var/lib/kubelet/pods/6b082e35-1549-4590-93f3-2713bfdda109/volumes" Nov 03 21:03:09.195261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:09.195229 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:09.196237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:09.196217 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:03:15.698395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.687337 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s"] Nov 03 21:03:15.698395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.689196 4227 cpu_manager.go:414] "containerMap: removing container" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" Nov 03 21:03:15.698395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.689220 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" Nov 03 21:03:15.698395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.689393 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b082e35-1549-4590-93f3-2713bfdda109" containerName="istio-proxy" Nov 03 21:03:15.711261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.711225 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:15.731697 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.728926 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s"] Nov 03 21:03:15.770062 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.770026 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3ca125\"" Nov 03 21:03:15.786479 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.772019 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3ca125-dockercfg-2rwqr\"" Nov 03 21:03:15.815182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.807819 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8f9\" (UniqueName: \"kubernetes.io/projected/2c642461-3a57-4cca-89d1-20caf9f69d44-kube-api-access-4w8f9\") pod \"payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:15.815182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.807903 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c642461-3a57-4cca-89d1-20caf9f69d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:15.828151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.816270 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j"] Nov 03 21:03:15.828151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.822487 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:15.884001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.883247 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j"] Nov 03 21:03:15.923238 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.916121 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8f9\" (UniqueName: \"kubernetes.io/projected/2c642461-3a57-4cca-89d1-20caf9f69d44-kube-api-access-4w8f9\") pod \"payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:15.923238 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:15.916241 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c642461-3a57-4cca-89d1-20caf9f69d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:16.031282 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.026396 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8f9\" (UniqueName: \"kubernetes.io/projected/2c642461-3a57-4cca-89d1-20caf9f69d44-kube-api-access-4w8f9\") pod \"payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:16.046215 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.039303 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c642461-3a57-4cca-89d1-20caf9f69d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:16.071192 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.049607 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8cg\" (UniqueName: \"kubernetes.io/projected/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-kube-api-access-bh8cg\") pod \"payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:16.071192 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.049741 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:16.150927 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.150885 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:16.151101 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.151088 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8cg\" (UniqueName: \"kubernetes.io/projected/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-kube-api-access-bh8cg\") pod \"payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:16.151699 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.151673 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:16.204209 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.204175 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8cg\" (UniqueName: \"kubernetes.io/projected/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-kube-api-access-bh8cg\") pod \"payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:16.339217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.339139 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:16.448321 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.447456 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:16.714603 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.714571 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s"] Nov 03 21:03:16.729248 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:03:16.729218 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c642461_3a57_4cca_89d1_20caf9f69d44.slice/crio-41e81562614c3938dcf928db8f9e969174578b771fceb306a3b1c24fd3ceddd6 WatchSource:0}: Error finding container 41e81562614c3938dcf928db8f9e969174578b771fceb306a3b1c24fd3ceddd6: Status 404 returned error can't find the container with id 41e81562614c3938dcf928db8f9e969174578b771fceb306a3b1c24fd3ceddd6 Nov 03 21:03:16.898777 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:03:16.898743 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec8860f_fba0_4a16_b9c6_c155a51f9a2c.slice/crio-adc1993bf3ac2578f81699a946ad38d26ee0f65687406c019604f916d51634dd WatchSource:0}: Error finding container adc1993bf3ac2578f81699a946ad38d26ee0f65687406c019604f916d51634dd: Status 404 returned error can't find the container with id adc1993bf3ac2578f81699a946ad38d26ee0f65687406c019604f916d51634dd Nov 03 21:03:16.902647 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:16.902618 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j"] Nov 03 21:03:17.267503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.267463 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" event={"ID":"2c642461-3a57-4cca-89d1-20caf9f69d44","Type":"ContainerStarted","Data":"34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb"} Nov 03 21:03:17.267753 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.267729 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:17.267827 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.267777 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" event={"ID":"2c642461-3a57-4cca-89d1-20caf9f69d44","Type":"ContainerStarted","Data":"41e81562614c3938dcf928db8f9e969174578b771fceb306a3b1c24fd3ceddd6"} Nov 03 21:03:17.269465 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.269436 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" event={"ID":"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c","Type":"ContainerStarted","Data":"21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8"} Nov 03 21:03:17.269636 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.269473 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" event={"ID":"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c","Type":"ContainerStarted","Data":"adc1993bf3ac2578f81699a946ad38d26ee0f65687406c019604f916d51634dd"} Nov 03 21:03:17.269636 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.269542 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:17.376440 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.376331 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" podStartSLOduration=2.376315562 podStartE2EDuration="2.376315562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:17.373829214 +0000 UTC m=+2064.610807088" watchObservedRunningTime="2026-11-03 21:03:17.376315562 +0000 UTC m=+2064.613293441" Nov 03 21:03:17.486229 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:17.486134 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" podStartSLOduration=2.486114945 podStartE2EDuration="2.486114945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:17.479260894 +0000 UTC m=+2064.716238769" watchObservedRunningTime="2026-11-03 21:03:17.486114945 +0000 UTC m=+2064.723092820" Nov 03 21:03:26.084192 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:03:26.084142 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:03:28.338192 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:28.336583 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:03:28.349182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:28.344372 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:03:39.122284 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.122249 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5"] Nov 03 21:03:39.128882 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.128857 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.198686 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.198648 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.198686 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.198689 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.198925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.198719 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.198925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.198842 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.199029 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.198961 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.199029 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.199007 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.199214 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.199194 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.199274 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.199250 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.199402 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.199382 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkcfq\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-kube-api-access-kkcfq\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301238 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301194 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301416 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301257 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301416 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301292 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301416 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301344 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301416 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301408 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301452 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301564 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301615 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301677 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301735 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301748 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkcfq\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-kube-api-access-kkcfq\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.301981 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301958 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.302040 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.301997 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.302319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.302293 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.303737 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.303712 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.304267 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.304250 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.318915 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.318879 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5"] Nov 03 21:03:39.356060 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.355547 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.356715 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.356681 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkcfq\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-kube-api-access-kkcfq\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.444429 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.444397 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:39.883680 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.883596 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5"] Nov 03 21:03:39.887451 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:03:39.887416 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ffea4b_bef0_4e2c_b7e0_b83e84bd9b1e.slice/crio-a1e9640cd381b54869a3a9495324606126c010d5863be53f8b34b177bc5aa422 WatchSource:0}: Error finding container a1e9640cd381b54869a3a9495324606126c010d5863be53f8b34b177bc5aa422: Status 404 returned error can't find the container with id a1e9640cd381b54869a3a9495324606126c010d5863be53f8b34b177bc5aa422 Nov 03 21:03:39.890802 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.890763 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:39.890893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.890851 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:39.890933 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:39.890893 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:40.485236 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.485193 4227 generic.go:356] "Generic (PLEG): container finished" podID="08577ba9-52bb-4c8c-8879-63deb9968296" containerID="7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53" exitCode=0 Nov 03 21:03:40.485643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.485395 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" event={"ID":"08577ba9-52bb-4c8c-8879-63deb9968296","Type":"ContainerDied","Data":"7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53"} Nov 03 21:03:40.485901 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.485882 4227 scope.go:122] "RemoveContainer" containerID="7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53" Nov 03 21:03:40.488040 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.488007 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:40.488181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.488089 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:40.488181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.488136 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:03:40.489006 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.488980 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" event={"ID":"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e","Type":"ContainerStarted","Data":"8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652"} Nov 03 21:03:40.489154 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.489137 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" event={"ID":"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e","Type":"ContainerStarted","Data":"a1e9640cd381b54869a3a9495324606126c010d5863be53f8b34b177bc5aa422"} Nov 03 21:03:40.490639 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.490621 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:03:40.716721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:40.716652 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podStartSLOduration=1.7166339609999999 podStartE2EDuration="1.716633961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:40.715996275 +0000 UTC m=+2087.952974152" watchObservedRunningTime="2026-11-03 21:03:40.716633961 +0000 UTC m=+2087.953611832" Nov 03 21:03:41.444837 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:41.444803 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:41.446133 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:41.446107 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:03:41.446307 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:41.446195 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:03:41.498542 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:41.498501 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" event={"ID":"08577ba9-52bb-4c8c-8879-63deb9968296","Type":"ContainerStarted","Data":"4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee"} Nov 03 21:03:41.559187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:41.555644 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf"] Nov 03 21:03:41.559187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:41.555878 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" containerID="cri-o://d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de" gracePeriod=30 Nov 03 21:03:42.144183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:42.144093 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:03:42.145208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:42.145182 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 21:03:42.145327 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:42.145231 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 21:03:42.445608 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:42.445469 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:03:42.445608 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:42.445554 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:03:43.143914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:43.143874 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 21:03:43.144414 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:43.143932 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 21:03:43.445035 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:43.444995 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:03:43.445258 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:43.445053 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:03:44.143742 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:44.143708 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 21:03:44.143920 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:44.143758 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 21:03:44.445571 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:44.445529 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:03:44.445956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:44.445590 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:03:45.025319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.025280 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx"] Nov 03 21:03:45.025538 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.025504 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" containerID="cri-o://734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660" gracePeriod=30 Nov 03 21:03:45.108518 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.108418 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg"] Nov 03 21:03:45.125201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.119830 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" containerID="cri-o://56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b" gracePeriod=30 Nov 03 21:03:45.183304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.167742 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 21:03:45.183304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.167823 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 21:03:45.445589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.445558 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:03:45.445954 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:45.445607 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:03:46.143510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:46.143482 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 21:03:46.143728 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:46.143707 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 21:03:46.445588 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:46.445557 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 21:03:46.446100 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:46.445608 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 21:03:46.889873 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:46.889820 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:03:46.890197 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:46.890142 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:03:46.942018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:46.941995 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 21:03:47.039402 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039318 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a674663-b61a-44bf-be2e-540c627e852c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a674663-b61a-44bf-be2e-540c627e852c-istiod-ca-cert\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.039402 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039397 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a674663-b61a-44bf-be2e-540c627e852c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a674663-b61a-44bf-be2e-540c627e852c-istio-podinfo\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.039637 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039450 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-credential-socket\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.039637 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039603 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-istio-token\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-istio-token\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.039771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039679 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a674663-b61a-44bf-be2e-540c627e852c-istiod-ca-cert" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:47.039771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039686 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-data\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.039892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039772 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-kube-api-access-zk7s7\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-kube-api-access-zk7s7\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.039892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039832 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-socket\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.040004 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039932 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-data" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:47.040004 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.039989 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-certs\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.040117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.040045 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-envoy\") pod \"5a674663-b61a-44bf-be2e-540c627e852c\" (UID: \"5a674663-b61a-44bf-be2e-540c627e852c\") " Nov 03 21:03:47.040117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.040051 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-socket" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:47.040266 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.040204 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-credential-socket" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:47.040880 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.040796 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.040880 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.040824 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.040880 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.040840 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a674663-b61a-44bf-be2e-540c627e852c-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.040880 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.040855 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.041191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.041039 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-certs" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:47.042251 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.042211 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-kube-api-access-zk7s7" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "kube-api-access-zk7s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:47.042362 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.042296 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-istio-token" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:47.042598 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.042570 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a674663-b61a-44bf-be2e-540c627e852c-istio-podinfo" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:47.043074 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.043053 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-envoy" pod "5a674663-b61a-44bf-be2e-540c627e852c" (UID: "5a674663-b61a-44bf-be2e-540c627e852c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:47.142295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.142257 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk7s7\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-kube-api-access-zk7s7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.142295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.142292 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.142503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.142309 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a674663-b61a-44bf-be2e-540c627e852c-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.142503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.142322 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a674663-b61a-44bf-be2e-540c627e852c-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.142503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.142334 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a674663-b61a-44bf-be2e-540c627e852c-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.207577 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.207542 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:03:47.208231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.208198 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:03:47.209023 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.209003 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:03:47.450334 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.450294 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:47.451018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.450994 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:47.451699 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.451673 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:03:47.554317 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.554285 4227 generic.go:356] "Generic (PLEG): container finished" podID="5a674663-b61a-44bf-be2e-540c627e852c" containerID="d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de" exitCode=0 Nov 03 21:03:47.554494 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.554351 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" Nov 03 21:03:47.554494 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.554361 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" event={"ID":"5a674663-b61a-44bf-be2e-540c627e852c","Type":"ContainerDied","Data":"d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de"} Nov 03 21:03:47.554494 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.554424 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf" event={"ID":"5a674663-b61a-44bf-be2e-540c627e852c","Type":"ContainerDied","Data":"c2c9c0ed7651abf14454ad5b549e061750e1087ab1eeeed30de3758e394fa6ab"} Nov 03 21:03:47.554494 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.554454 4227 scope.go:122] "RemoveContainer" containerID="d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de" Nov 03 21:03:47.566003 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.565977 4227 scope.go:122] "RemoveContainer" containerID="d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de" Nov 03 21:03:47.566286 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:03:47.566266 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de\": container with ID starting with d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de not found: ID does not exist" containerID="d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de" Nov 03 21:03:47.566345 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.566295 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de"} err="failed to get container status \"d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de\": rpc error: code = NotFound desc = could not find container \"d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de\": container with ID starting with d775d533313fac4876853fed4f637d6d960f6271df8a0f81d95b3444d9cb28de not found: ID does not exist" Nov 03 21:03:47.820616 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.820543 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf"] Nov 03 21:03:47.829180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.826789 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-rmdcf"] Nov 03 21:03:47.987201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.987139 4227 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:03:47.987354 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:47.987225 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:03:48.901238 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:48.901206 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a674663-b61a-44bf-be2e-540c627e852c" path="/var/lib/kubelet/pods/5a674663-b61a-44bf-be2e-540c627e852c/volumes" Nov 03 21:03:50.318582 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.318549 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 21:03:50.385436 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385217 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-podinfo\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385436 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385336 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-kube-api-access-2h5v4\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-kube-api-access-2h5v4\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385436 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385419 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-credential-socket\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385487 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-socket\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385598 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-envoy\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385659 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-token\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-token\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385838 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385695 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-certs\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385838 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385767 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-data\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.385838 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.385801 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istiod-ca-cert\") pod \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\" (UID: \"6751b5ae-fbb7-4529-bbb5-be422bdeb53e\") " Nov 03 21:03:50.388467 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.388425 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-credential-socket" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.388588 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.388477 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istiod-ca-cert" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:50.389205 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.388909 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-data" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.389205 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.388979 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-certs" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.389205 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.389087 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-socket" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.392185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.391639 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-envoy" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.392185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.391919 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-token" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:50.392185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.391987 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-podinfo" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:50.393109 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.393072 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-kube-api-access-2h5v4" pod "6751b5ae-fbb7-4529-bbb5-be422bdeb53e" (UID: "6751b5ae-fbb7-4529-bbb5-be422bdeb53e"). InnerVolumeSpecName "kube-api-access-2h5v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:50.398196 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.398174 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 21:03:50.488777 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.488736 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-data\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.488961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.488799 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-podinfo\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.488961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.488861 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-socket\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.488961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.488909 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-token\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-token\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.489136 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.489045 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-data" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.489136 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.489054 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-certs\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.489256 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.489135 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-credential-socket\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.489848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.489253 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-kube-api-access-mtzt7\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-kube-api-access-mtzt7\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.489848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.489356 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88970ce2-dfab-4462-b5d9-3797ea5d2072-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88970ce2-dfab-4462-b5d9-3797ea5d2072-istiod-ca-cert\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.489848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.489445 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-envoy\") pod \"88970ce2-dfab-4462-b5d9-3797ea5d2072\" (UID: \"88970ce2-dfab-4462-b5d9-3797ea5d2072\") " Nov 03 21:03:50.489848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.489489 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-certs" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490504 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490528 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490544 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490559 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490576 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490592 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490607 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490621 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5v4\" (UniqueName: \"kubernetes.io/projected/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-kube-api-access-2h5v4\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490635 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490648 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.490664 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6751b5ae-fbb7-4529-bbb5-be422bdeb53e-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.491291 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-credential-socket" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.491349 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-socket" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.491528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.491491 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88970ce2-dfab-4462-b5d9-3797ea5d2072-istiod-ca-cert" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:50.495510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.495415 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-podinfo" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:50.495510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.495484 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-envoy" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:50.495510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.495497 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-kube-api-access-mtzt7" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "kube-api-access-mtzt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:50.496700 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.496676 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-token" pod "88970ce2-dfab-4462-b5d9-3797ea5d2072" (UID: "88970ce2-dfab-4462-b5d9-3797ea5d2072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:50.575757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.575674 4227 generic.go:356] "Generic (PLEG): container finished" podID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerID="734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660" exitCode=0 Nov 03 21:03:50.575757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.575740 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" Nov 03 21:03:50.575966 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.575771 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" event={"ID":"6751b5ae-fbb7-4529-bbb5-be422bdeb53e","Type":"ContainerDied","Data":"734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660"} Nov 03 21:03:50.575966 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.575832 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx" event={"ID":"6751b5ae-fbb7-4529-bbb5-be422bdeb53e","Type":"ContainerDied","Data":"48a05f0e71fb2df17ceb5ff38d196381e4c07875ede5ab640eba3f456ac67087"} Nov 03 21:03:50.575966 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.575861 4227 scope.go:122] "RemoveContainer" containerID="734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660" Nov 03 21:03:50.577303 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.577279 4227 generic.go:356] "Generic (PLEG): container finished" podID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerID="56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b" exitCode=0 Nov 03 21:03:50.577430 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.577414 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" event={"ID":"88970ce2-dfab-4462-b5d9-3797ea5d2072","Type":"ContainerDied","Data":"56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b"} Nov 03 21:03:50.577498 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.577454 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" event={"ID":"88970ce2-dfab-4462-b5d9-3797ea5d2072","Type":"ContainerDied","Data":"83716b4583b366a763c77f71804565c28ec15f2adfc28f7112bd3709ebe29100"} Nov 03 21:03:50.577561 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.577520 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg" Nov 03 21:03:50.596432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.592970 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtzt7\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-kube-api-access-mtzt7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.596432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.593003 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88970ce2-dfab-4462-b5d9-3797ea5d2072-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.596432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.593052 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.596432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.593067 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.596432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.593081 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.596432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.593094 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88970ce2-dfab-4462-b5d9-3797ea5d2072-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.596432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.593140 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88970ce2-dfab-4462-b5d9-3797ea5d2072-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.607278 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.607250 4227 scope.go:122] "RemoveContainer" containerID="734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660" Nov 03 21:03:50.607680 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:03:50.607653 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660\": container with ID starting with 734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660 not found: ID does not exist" containerID="734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660" Nov 03 21:03:50.607794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.607689 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660"} err="failed to get container status \"734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660\": rpc error: code = NotFound desc = could not find container \"734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660\": container with ID starting with 734d33c7256afa3cc4759e8b0bfa6f772f8200df7b3462e9abcdb77d97968660 not found: ID does not exist" Nov 03 21:03:50.607794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.607708 4227 scope.go:122] "RemoveContainer" containerID="56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b" Nov 03 21:03:50.619456 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.619403 4227 scope.go:122] "RemoveContainer" containerID="56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b" Nov 03 21:03:50.619774 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:03:50.619745 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b\": container with ID starting with 56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b not found: ID does not exist" containerID="56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b" Nov 03 21:03:50.619903 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.619782 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b"} err="failed to get container status \"56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b\": rpc error: code = NotFound desc = could not find container \"56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b\": container with ID starting with 56ab965496a76de14a3f38e633eaeac62ee3db887c24fdb336c6cacbe18fcc7b not found: ID does not exist" Nov 03 21:03:50.790169 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.790123 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg"] Nov 03 21:03:50.797261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.797230 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-82mfg"] Nov 03 21:03:50.949184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.942725 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" path="/var/lib/kubelet/pods/88970ce2-dfab-4462-b5d9-3797ea5d2072/volumes" Nov 03 21:03:50.967807 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.967776 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx"] Nov 03 21:03:50.968771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:50.968742 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-zgcgx"] Nov 03 21:03:52.903795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:52.903764 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" path="/var/lib/kubelet/pods/6751b5ae-fbb7-4529-bbb5-be422bdeb53e/volumes" Nov 03 21:03:53.135333 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:53.135302 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:03:53.140209 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:53.140149 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:03:53.491621 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:53.491590 4227 scope.go:122] "RemoveContainer" containerID="24cd5f0f735798be57f3250ba476b76fafb58c3c231fe1f6a9429a1e071acf19" Nov 03 21:03:56.243477 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:03:56.243424 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:03:56.661297 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:56.661262 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f"] Nov 03 21:03:56.661502 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:03:56.661466 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" containerID="cri-o://104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e" gracePeriod=30 Nov 03 21:04:00.149498 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:00.149462 4227 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:04:00.149964 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:00.149536 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:04:01.940593 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.940529 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 21:04:01.996283 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996244 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93442648-c09f-4e6d-a521-9d4ed002812a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93442648-c09f-4e6d-a521-9d4ed002812a-istiod-ca-cert\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996498 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996375 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-kube-api-access-gqgsm\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-kube-api-access-gqgsm\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996498 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996412 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-istio-token\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-istio-token\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996498 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996466 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-data\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996676 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996525 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-socket\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996676 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996588 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-credential-socket\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996676 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996662 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93442648-c09f-4e6d-a521-9d4ed002812a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93442648-c09f-4e6d-a521-9d4ed002812a-istio-podinfo\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996847 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996825 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-envoy\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.996906 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996859 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-data" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:01.996998 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.996968 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-certs\") pod \"93442648-c09f-4e6d-a521-9d4ed002812a\" (UID: \"93442648-c09f-4e6d-a521-9d4ed002812a\") " Nov 03 21:04:01.997353 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997320 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-credential-socket" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:01.997558 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997524 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-socket" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:01.997669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997556 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-certs" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:01.997843 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997821 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.997930 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997848 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.997930 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997862 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.997930 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997878 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.998093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.997958 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93442648-c09f-4e6d-a521-9d4ed002812a-istiod-ca-cert" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:01.999054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.999021 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-kube-api-access-gqgsm" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "kube-api-access-gqgsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:01.999054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.999039 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93442648-c09f-4e6d-a521-9d4ed002812a-istio-podinfo" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:04:01.999207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.999066 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-envoy" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:01.999207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:01.999102 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-istio-token" pod "93442648-c09f-4e6d-a521-9d4ed002812a" (UID: "93442648-c09f-4e6d-a521-9d4ed002812a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:02.099603 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.099563 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgsm\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-kube-api-access-gqgsm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:02.099603 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.099605 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93442648-c09f-4e6d-a521-9d4ed002812a-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:02.099884 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.099620 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93442648-c09f-4e6d-a521-9d4ed002812a-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:02.099884 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.099634 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93442648-c09f-4e6d-a521-9d4ed002812a-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:02.099884 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.099647 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93442648-c09f-4e6d-a521-9d4ed002812a-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:02.671296 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.671258 4227 generic.go:356] "Generic (PLEG): container finished" podID="93442648-c09f-4e6d-a521-9d4ed002812a" containerID="104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e" exitCode=0 Nov 03 21:04:02.671445 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.671316 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" Nov 03 21:04:02.671445 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.671318 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" event={"ID":"93442648-c09f-4e6d-a521-9d4ed002812a","Type":"ContainerDied","Data":"104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e"} Nov 03 21:04:02.671544 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.671451 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f" event={"ID":"93442648-c09f-4e6d-a521-9d4ed002812a","Type":"ContainerDied","Data":"b60d3b00f8a6b9bc42f56d010f657c7c98998ab2d45d5e76e88397df1d8d3b65"} Nov 03 21:04:02.671544 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.671480 4227 scope.go:122] "RemoveContainer" containerID="104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e" Nov 03 21:04:02.683563 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.683539 4227 scope.go:122] "RemoveContainer" containerID="104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e" Nov 03 21:04:02.683898 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:02.683878 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e\": container with ID starting with 104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e not found: ID does not exist" containerID="104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e" Nov 03 21:04:02.683975 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.683905 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e"} err="failed to get container status \"104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e\": rpc error: code = NotFound desc = could not find container \"104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e\": container with ID starting with 104da6514e2c24be262ea3b85624d161d005d57fa309c33bb2a5dc8dc359242e not found: ID does not exist" Nov 03 21:04:02.872734 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.872699 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f"] Nov 03 21:04:02.920271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:02.920233 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-jzg2f"] Nov 03 21:04:04.900932 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:04.900852 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" path="/var/lib/kubelet/pods/93442648-c09f-4e6d-a521-9d4ed002812a/volumes" Nov 03 21:04:08.599771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:08.599734 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4"] Nov 03 21:04:08.715591 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:08.715537 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464"] Nov 03 21:04:13.600785 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.600735 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" podUID="84d9eb92-19eb-436a-a7d7-515d9e22af7c" containerName="payload-processing" containerID="cri-o://082a76294b04780257d1bcef9aac20a29dd9fab087d08ed9f50bd744a7e77a9d" gracePeriod=25 Nov 03 21:04:13.716921 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.716880 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" podUID="1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" containerName="payload-pre-processing" containerID="cri-o://41f034dae5f66e4cd0c8fd07fb76f02baeb3551bcb1df9c2789c39f77b5fd961" gracePeriod=25 Nov 03 21:04:13.838902 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.838868 4227 generic.go:356] "Generic (PLEG): container finished" podID="84d9eb92-19eb-436a-a7d7-515d9e22af7c" containerID="082a76294b04780257d1bcef9aac20a29dd9fab087d08ed9f50bd744a7e77a9d" exitCode=0 Nov 03 21:04:13.839025 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.838942 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" event={"ID":"84d9eb92-19eb-436a-a7d7-515d9e22af7c","Type":"ContainerDied","Data":"082a76294b04780257d1bcef9aac20a29dd9fab087d08ed9f50bd744a7e77a9d"} Nov 03 21:04:13.840772 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.840752 4227 generic.go:356] "Generic (PLEG): container finished" podID="1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" containerID="41f034dae5f66e4cd0c8fd07fb76f02baeb3551bcb1df9c2789c39f77b5fd961" exitCode=0 Nov 03 21:04:13.840885 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.840825 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" event={"ID":"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d","Type":"ContainerDied","Data":"41f034dae5f66e4cd0c8fd07fb76f02baeb3551bcb1df9c2789c39f77b5fd961"} Nov 03 21:04:13.871553 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.871493 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:04:13.954948 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.954919 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d9eb92-19eb-436a-a7d7-515d9e22af7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9eb92-19eb-436a-a7d7-515d9e22af7c-plugins-config-volume\") pod \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " Nov 03 21:04:13.955206 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.955045 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d9eb92-19eb-436a-a7d7-515d9e22af7c-kube-api-access-5n4f4\" (UniqueName: \"kubernetes.io/projected/84d9eb92-19eb-436a-a7d7-515d9e22af7c-kube-api-access-5n4f4\") pod \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\" (UID: \"84d9eb92-19eb-436a-a7d7-515d9e22af7c\") " Nov 03 21:04:13.955393 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.955352 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d9eb92-19eb-436a-a7d7-515d9e22af7c-plugins-config-volume" pod "84d9eb92-19eb-436a-a7d7-515d9e22af7c" (UID: "84d9eb92-19eb-436a-a7d7-515d9e22af7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:13.955991 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.955967 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9eb92-19eb-436a-a7d7-515d9e22af7c-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:13.958128 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.958103 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d9eb92-19eb-436a-a7d7-515d9e22af7c-kube-api-access-5n4f4" pod "84d9eb92-19eb-436a-a7d7-515d9e22af7c" (UID: "84d9eb92-19eb-436a-a7d7-515d9e22af7c"). InnerVolumeSpecName "kube-api-access-5n4f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:13.996019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:13.995986 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:04:14.057010 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.056976 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-plugins-config-volume\") pod \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " Nov 03 21:04:14.057216 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.057079 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-kube-api-access-w7ss2\" (UniqueName: \"kubernetes.io/projected/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-kube-api-access-w7ss2\") pod \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\" (UID: \"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d\") " Nov 03 21:04:14.057470 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.057449 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-plugins-config-volume" pod "1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" (UID: "1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:14.057670 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.057655 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n4f4\" (UniqueName: \"kubernetes.io/projected/84d9eb92-19eb-436a-a7d7-515d9e22af7c-kube-api-access-5n4f4\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:14.057709 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.057673 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:14.059132 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.059110 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-kube-api-access-w7ss2" pod "1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" (UID: "1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d"). InnerVolumeSpecName "kube-api-access-w7ss2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:14.158593 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.158565 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ss2\" (UniqueName: \"kubernetes.io/projected/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d-kube-api-access-w7ss2\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:14.848100 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.848068 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" Nov 03 21:04:14.848614 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.848094 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4" event={"ID":"84d9eb92-19eb-436a-a7d7-515d9e22af7c","Type":"ContainerDied","Data":"414bc0c7f3cf42901d2e4d8605c2e9cd4c73dfc9dbfa3b60083301748c26217b"} Nov 03 21:04:14.848614 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.848181 4227 scope.go:122] "RemoveContainer" containerID="082a76294b04780257d1bcef9aac20a29dd9fab087d08ed9f50bd744a7e77a9d" Nov 03 21:04:14.854842 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.854815 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" event={"ID":"1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d","Type":"ContainerDied","Data":"e3471268832ae212a503dfc2c2d68b33d11da418e8e716fef51a05e97ab643a8"} Nov 03 21:04:14.855116 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.854919 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464" Nov 03 21:04:14.861323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:14.861302 4227 scope.go:122] "RemoveContainer" containerID="41f034dae5f66e4cd0c8fd07fb76f02baeb3551bcb1df9c2789c39f77b5fd961" Nov 03 21:04:15.003711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:15.003678 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4"] Nov 03 21:04:15.051827 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:15.051794 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6aaa6-549f64c9bc-kxzw4"] Nov 03 21:04:15.109709 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:15.109602 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464"] Nov 03 21:04:15.112543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:15.112513 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6aaa6-7764776dd7-vg464"] Nov 03 21:04:15.794090 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:15.794055 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn"] Nov 03 21:04:16.901007 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:16.900979 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" path="/var/lib/kubelet/pods/1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d/volumes" Nov 03 21:04:16.901515 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:16.901383 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d9eb92-19eb-436a-a7d7-515d9e22af7c" path="/var/lib/kubelet/pods/84d9eb92-19eb-436a-a7d7-515d9e22af7c/volumes" Nov 03 21:04:20.795066 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:20.795000 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" podUID="41292328-b7e4-4c4e-8bd9-e17ce0f867dc" containerName="payload-processing" containerID="cri-o://36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c" gracePeriod=25 Nov 03 21:04:21.169588 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.169559 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:04:21.314876 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.314834 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-plugins-config-volume\") pod \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " Nov 03 21:04:21.315040 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.314916 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-kube-api-access-t88p6\" (UniqueName: \"kubernetes.io/projected/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-kube-api-access-t88p6\") pod \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\" (UID: \"41292328-b7e4-4c4e-8bd9-e17ce0f867dc\") " Nov 03 21:04:21.315543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.315515 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-plugins-config-volume" pod "41292328-b7e4-4c4e-8bd9-e17ce0f867dc" (UID: "41292328-b7e4-4c4e-8bd9-e17ce0f867dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:21.317508 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.317482 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-kube-api-access-t88p6" pod "41292328-b7e4-4c4e-8bd9-e17ce0f867dc" (UID: "41292328-b7e4-4c4e-8bd9-e17ce0f867dc"). InnerVolumeSpecName "kube-api-access-t88p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:21.417185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.417128 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:21.417185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.417182 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t88p6\" (UniqueName: \"kubernetes.io/projected/41292328-b7e4-4c4e-8bd9-e17ce0f867dc-kube-api-access-t88p6\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:21.909171 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.909114 4227 generic.go:356] "Generic (PLEG): container finished" podID="41292328-b7e4-4c4e-8bd9-e17ce0f867dc" containerID="36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c" exitCode=0 Nov 03 21:04:21.909667 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.909197 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" Nov 03 21:04:21.909667 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.909210 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" event={"ID":"41292328-b7e4-4c4e-8bd9-e17ce0f867dc","Type":"ContainerDied","Data":"36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c"} Nov 03 21:04:21.909667 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.909272 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn" event={"ID":"41292328-b7e4-4c4e-8bd9-e17ce0f867dc","Type":"ContainerDied","Data":"244c9a5efb50c077b23b07ff8702b8f80613cdedfb4847e67e58f5c905b8cd46"} Nov 03 21:04:21.909667 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.909294 4227 scope.go:122] "RemoveContainer" containerID="36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c" Nov 03 21:04:21.919995 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.919975 4227 scope.go:122] "RemoveContainer" containerID="36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c" Nov 03 21:04:21.921135 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:21.921112 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c\": container with ID starting with 36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c not found: ID does not exist" containerID="36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c" Nov 03 21:04:21.921252 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:21.921146 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c"} err="failed to get container status \"36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c\": rpc error: code = NotFound desc = could not find container \"36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c\": container with ID starting with 36d114de870b86ac4d0674b87beaf0c4b5b8209aa3097d7c3da9fd1a8ce4104c not found: ID does not exist" Nov 03 21:04:22.068186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:22.065386 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn"] Nov 03 21:04:22.123026 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:22.121023 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-20d356-68cf7d5977-vdnjn"] Nov 03 21:04:22.903346 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:22.903316 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41292328-b7e4-4c4e-8bd9-e17ce0f867dc" path="/var/lib/kubelet/pods/41292328-b7e4-4c4e-8bd9-e17ce0f867dc/volumes" Nov 03 21:04:26.083774 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:26.083738 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:04:33.926831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.926800 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g"] Nov 03 21:04:33.927556 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927534 4227 cpu_manager.go:414] "containerMap: removing container" podUID="1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" containerName="payload-pre-processing" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927558 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" containerName="payload-pre-processing" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927576 4227 cpu_manager.go:414] "containerMap: removing container" podUID="41292328-b7e4-4c4e-8bd9-e17ce0f867dc" containerName="payload-processing" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927583 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41292328-b7e4-4c4e-8bd9-e17ce0f867dc" containerName="payload-processing" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927604 4227 cpu_manager.go:414] "containerMap: removing container" podUID="84d9eb92-19eb-436a-a7d7-515d9e22af7c" containerName="payload-processing" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927612 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d9eb92-19eb-436a-a7d7-515d9e22af7c" containerName="payload-processing" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927649 4227 cpu_manager.go:414] "containerMap: removing container" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927657 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927684 4227 cpu_manager.go:414] "containerMap: removing container" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927691 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927712 4227 cpu_manager.go:414] "containerMap: removing container" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" Nov 03 21:04:33.927721 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927719 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927733 4227 cpu_manager.go:414] "containerMap: removing container" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927740 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927877 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="6751b5ae-fbb7-4529-bbb5-be422bdeb53e" containerName="istio-proxy" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927890 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="88970ce2-dfab-4462-b5d9-3797ea5d2072" containerName="istio-proxy" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927905 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="93442648-c09f-4e6d-a521-9d4ed002812a" containerName="istio-proxy" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927914 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e970a1f-cb48-4d6c-aa8e-7ff6354f3d1d" containerName="payload-pre-processing" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927925 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="41292328-b7e4-4c4e-8bd9-e17ce0f867dc" containerName="payload-processing" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927935 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d9eb92-19eb-436a-a7d7-515d9e22af7c" containerName="payload-processing" Nov 03 21:04:33.928262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.927952 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a674663-b61a-44bf-be2e-540c627e852c" containerName="istio-proxy" Nov 03 21:04:33.932848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.932830 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:33.982473 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:33.982434 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c571ef-serving-cert\"" Nov 03 21:04:34.011091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.011050 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce8e3d-bf70-46a2-b705-f52e42403c79-openshift-service-ca\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.011284 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.011117 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-metrics-certs\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.011284 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.011235 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.011410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.011308 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vvmq\" (UniqueName: \"kubernetes.io/projected/b8ce8e3d-bf70-46a2-b705-f52e42403c79-kube-api-access-6vvmq\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.031527 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.031499 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g"] Nov 03 21:04:34.113075 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.113039 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-metrics-certs\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.113286 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.113242 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.113382 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:34.113364 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-serving-cert: secret "maas-api-e2e-api-b-c571ef-serving-cert" not found Nov 03 21:04:34.113449 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.113376 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vvmq\" (UniqueName: \"kubernetes.io/projected/b8ce8e3d-bf70-46a2-b705-f52e42403c79-kube-api-access-6vvmq\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.113449 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:34.113446 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls podName:b8ce8e3d-bf70-46a2-b705-f52e42403c79 nodeName:}" failed. No retries permitted until 2026-11-03 21:04:34.61342145 +0000 UTC m=+2141.850399305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls") pod "maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" (UID: "b8ce8e3d-bf70-46a2-b705-f52e42403c79") : secret "maas-api-e2e-api-b-c571ef-serving-cert" not found Nov 03 21:04:34.113615 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.113596 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce8e3d-bf70-46a2-b705-f52e42403c79-openshift-service-ca\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.114323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.114298 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce8e3d-bf70-46a2-b705-f52e42403c79-openshift-service-ca\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.115749 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.115727 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-metrics-certs\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.168543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.168505 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vvmq\" (UniqueName: \"kubernetes.io/projected/b8ce8e3d-bf70-46a2-b705-f52e42403c79-kube-api-access-6vvmq\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.624127 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.624094 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.626611 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.626589 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls\") pod \"maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:34.847738 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:34.847704 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:35.229691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:35.229612 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g"] Nov 03 21:04:35.235979 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:04:35.235951 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ce8e3d_bf70_46a2_b705_f52e42403c79.slice/crio-2628451ca8ea20aaed726812abfc3b60e3d4f4a86b440e510dc3d1a5e5fe17d0 WatchSource:0}: Error finding container 2628451ca8ea20aaed726812abfc3b60e3d4f4a86b440e510dc3d1a5e5fe17d0: Status 404 returned error can't find the container with id 2628451ca8ea20aaed726812abfc3b60e3d4f4a86b440e510dc3d1a5e5fe17d0 Nov 03 21:04:35.237368 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:35.237347 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:04:36.002222 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:36.002187 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" event={"ID":"b8ce8e3d-bf70-46a2-b705-f52e42403c79","Type":"ContainerStarted","Data":"2628451ca8ea20aaed726812abfc3b60e3d4f4a86b440e510dc3d1a5e5fe17d0"} Nov 03 21:04:37.010048 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:37.010009 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" event={"ID":"b8ce8e3d-bf70-46a2-b705-f52e42403c79","Type":"ContainerStarted","Data":"efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec"} Nov 03 21:04:37.010476 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:37.010221 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:37.120984 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:37.120925 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" podStartSLOduration=3.101465278 podStartE2EDuration="4.12091046s" totalImagesPullingTime="1.019445182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:37.117035067 +0000 UTC m=+2144.354012941" watchObservedRunningTime="2026-11-03 21:04:37.12091046 +0000 UTC m=+2144.357888334" Nov 03 21:04:43.022043 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:43.022012 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:04:44.255014 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.254979 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z"] Nov 03 21:04:44.255473 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.255202 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" podUID="fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" containerName="maas-api" containerID="cri-o://f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7" gracePeriod=30 Nov 03 21:04:44.712585 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.712552 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 21:04:44.821182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.807998 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-openshift-service-ca\") pod \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " Nov 03 21:04:44.821182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.815412 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-openshift-service-ca" pod "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" (UID: "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:44.821182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.815547 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-kube-api-access-l24wc\" (UniqueName: \"kubernetes.io/projected/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-kube-api-access-l24wc\") pod \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " Nov 03 21:04:44.821182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.815743 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls\") pod \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " Nov 03 21:04:44.821182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.815803 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-metrics-certs\") pod \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\" (UID: \"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0\") " Nov 03 21:04:44.821182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.816668 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:44.838189 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.837376 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-metrics-certs" pod "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" (UID: "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:44.873180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.862505 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-kube-api-access-l24wc" pod "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" (UID: "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0"). InnerVolumeSpecName "kube-api-access-l24wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:44.933261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.888711 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls" pod "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" (UID: "fdea3f24-37ab-4bff-ad3f-a4b221fa37f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:44.933261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.920502 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24wc\" (UniqueName: \"kubernetes.io/projected/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-kube-api-access-l24wc\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:44.933261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.920539 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:44.933261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:44.920555 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:45.065090 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.065044 4227 generic.go:356] "Generic (PLEG): container finished" podID="fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" containerID="f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7" exitCode=0 Nov 03 21:04:45.065301 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.065112 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" Nov 03 21:04:45.065301 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.065107 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" event={"ID":"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0","Type":"ContainerDied","Data":"f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7"} Nov 03 21:04:45.065301 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.065230 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z" event={"ID":"fdea3f24-37ab-4bff-ad3f-a4b221fa37f0","Type":"ContainerDied","Data":"68e107fca0df24379e1d8d83cf0320c8e9980c6b65dea7482d3e25801653a634"} Nov 03 21:04:45.065301 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.065254 4227 scope.go:122] "RemoveContainer" containerID="f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7" Nov 03 21:04:45.075231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.075201 4227 scope.go:122] "RemoveContainer" containerID="f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7" Nov 03 21:04:45.075542 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:45.075522 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7\": container with ID starting with f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7 not found: ID does not exist" containerID="f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7" Nov 03 21:04:45.075604 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.075551 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7"} err="failed to get container status \"f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7\": rpc error: code = NotFound desc = could not find container \"f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7\": container with ID starting with f50a77ca1005ab80705c5c80baf634002bb9fb52c5267b4631119e5634d64cd7 not found: ID does not exist" Nov 03 21:04:45.218383 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.218308 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z"] Nov 03 21:04:45.221115 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:45.221091 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-b6d800-56996455c7-w584z"] Nov 03 21:04:46.043812 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.043778 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw"] Nov 03 21:04:46.044365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.044349 4227 cpu_manager.go:414] "containerMap: removing container" podUID="fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" containerName="maas-api" Nov 03 21:04:46.044413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.044366 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" containerName="maas-api" Nov 03 21:04:46.044504 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.044494 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" containerName="maas-api" Nov 03 21:04:46.052200 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.052174 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.109180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.102821 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw"] Nov 03 21:04:46.136510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.136462 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzgn\" (UniqueName: \"kubernetes.io/projected/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-kube-api-access-jrzgn\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.136715 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.136586 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-openshift-service-ca\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.136715 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.136618 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-metrics-certs\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.136715 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.136673 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.143764 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.143735 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3ca125-serving-cert\"" Nov 03 21:04:46.237961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.237919 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-openshift-service-ca\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.238145 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.237972 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-metrics-certs\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.238145 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.238032 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.238300 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.238194 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzgn\" (UniqueName: \"kubernetes.io/projected/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-kube-api-access-jrzgn\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.238424 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:46.238386 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-serving-cert: secret "maas-api-e2e-api-b-3ca125-serving-cert" not found Nov 03 21:04:46.238553 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:46.238461 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls podName:1f0c46b1-d75f-4f7d-b10d-5899d1770e64 nodeName:}" failed. No retries permitted until 2026-11-03 21:04:46.738442543 +0000 UTC m=+2153.975420409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls") pod "maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" (UID: "1f0c46b1-d75f-4f7d-b10d-5899d1770e64") : secret "maas-api-e2e-api-b-3ca125-serving-cert" not found Nov 03 21:04:46.238624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.238564 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-openshift-service-ca\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.240398 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.240376 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-metrics-certs\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.292530 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.292495 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzgn\" (UniqueName: \"kubernetes.io/projected/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-kube-api-access-jrzgn\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.745580 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.745548 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.748120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.748091 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls\") pod \"maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:46.901285 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.901253 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdea3f24-37ab-4bff-ad3f-a4b221fa37f0" path="/var/lib/kubelet/pods/fdea3f24-37ab-4bff-ad3f-a4b221fa37f0/volumes" Nov 03 21:04:46.968243 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:46.968206 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:47.373393 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:47.373314 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw"] Nov 03 21:04:47.378905 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:04:47.378878 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0c46b1_d75f_4f7d_b10d_5899d1770e64.slice/crio-dac586c84a4366a12a4ff37dee8ec631c02b7ed3dcd1bd73ee36715d84576760 WatchSource:0}: Error finding container dac586c84a4366a12a4ff37dee8ec631c02b7ed3dcd1bd73ee36715d84576760: Status 404 returned error can't find the container with id dac586c84a4366a12a4ff37dee8ec631c02b7ed3dcd1bd73ee36715d84576760 Nov 03 21:04:48.090221 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:48.090182 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" event={"ID":"1f0c46b1-d75f-4f7d-b10d-5899d1770e64","Type":"ContainerStarted","Data":"dac586c84a4366a12a4ff37dee8ec631c02b7ed3dcd1bd73ee36715d84576760"} Nov 03 21:04:49.101022 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:49.100974 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" event={"ID":"1f0c46b1-d75f-4f7d-b10d-5899d1770e64","Type":"ContainerStarted","Data":"de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f"} Nov 03 21:04:49.101566 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:49.101177 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:49.209519 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:49.209469 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" podStartSLOduration=2.297618057 podStartE2EDuration="3.209453043s" totalImagesPullingTime="911.834986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:49.207619249 +0000 UTC m=+2156.444597122" watchObservedRunningTime="2026-11-03 21:04:49.209453043 +0000 UTC m=+2156.446430917" Nov 03 21:04:54.108927 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:54.108893 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2"] Nov 03 21:04:54.109448 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:54.109146 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" containerID="cri-o://a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d" gracePeriod=30 Nov 03 21:04:55.115662 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:55.115633 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:04:55.484546 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:55.484506 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:04:55.484713 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:55.484578 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:04:56.082371 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:04:56.082333 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:04:59.387589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.387529 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 21:04:59.550779 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.550743 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbea0aa-e253-4668-bdb9-faa486867e9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbea0aa-e253-4668-bdb9-faa486867e9c-istiod-ca-cert\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.551089 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.551066 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-certs\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.551304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.551286 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-socket\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.551445 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.551431 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-envoy\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.551569 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.551554 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-kube-api-access-gtz9b\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-kube-api-access-gtz9b\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.551682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.551668 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-data\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.551923 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.551908 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-podinfo\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.552173 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.552141 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-token\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-token\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.552312 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.552297 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-credential-socket\") pod \"ddbea0aa-e253-4668-bdb9-faa486867e9c\" (UID: \"ddbea0aa-e253-4668-bdb9-faa486867e9c\") " Nov 03 21:04:59.553752 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.553728 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-credential-socket" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:59.554271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.554250 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbea0aa-e253-4668-bdb9-faa486867e9c-istiod-ca-cert" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:59.554619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.554594 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-certs" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:59.554795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.554776 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-socket" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:59.556735 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.556710 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-data" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:59.563827 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.563767 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-envoy" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:59.576581 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.576549 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-token" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:59.576719 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.576535 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-podinfo" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:04:59.576795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.576754 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-kube-api-access-gtz9b" pod "ddbea0aa-e253-4668-bdb9-faa486867e9c" (UID: "ddbea0aa-e253-4668-bdb9-faa486867e9c"). InnerVolumeSpecName "kube-api-access-gtz9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:59.655031 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655003 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655031 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655033 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655043 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655052 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbea0aa-e253-4668-bdb9-faa486867e9c-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655062 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655071 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655080 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655089 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtz9b\" (UniqueName: \"kubernetes.io/projected/ddbea0aa-e253-4668-bdb9-faa486867e9c-kube-api-access-gtz9b\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:59.655295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:04:59.655098 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbea0aa-e253-4668-bdb9-faa486867e9c-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.176507 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.176470 4227 generic.go:356] "Generic (PLEG): container finished" podID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerID="a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d" exitCode=0 Nov 03 21:05:00.176687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.176548 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" Nov 03 21:05:00.176687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.176562 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" event={"ID":"ddbea0aa-e253-4668-bdb9-faa486867e9c","Type":"ContainerDied","Data":"a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d"} Nov 03 21:05:00.176687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.176617 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2" event={"ID":"ddbea0aa-e253-4668-bdb9-faa486867e9c","Type":"ContainerDied","Data":"9230ba9a8e899989f66c0d86a46020dd303885890fde0eb35188dc5b3949ab09"} Nov 03 21:05:00.176687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.176645 4227 scope.go:122] "RemoveContainer" containerID="a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d" Nov 03 21:05:00.187111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.187087 4227 scope.go:122] "RemoveContainer" containerID="15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548" Nov 03 21:05:00.197771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.197750 4227 scope.go:122] "RemoveContainer" containerID="a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d" Nov 03 21:05:00.198074 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:00.198052 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d\": container with ID starting with a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d not found: ID does not exist" containerID="a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d" Nov 03 21:05:00.198142 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.198085 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d"} err="failed to get container status \"a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d\": rpc error: code = NotFound desc = could not find container \"a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d\": container with ID starting with a1cbe567a3123962487d67bea8a3f782cd6ec87581bcbfdaa377a295a2a4096d not found: ID does not exist" Nov 03 21:05:00.198142 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.198105 4227 scope.go:122] "RemoveContainer" containerID="15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548" Nov 03 21:05:00.198629 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:00.198607 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548\": container with ID starting with 15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548 not found: ID does not exist" containerID="15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548" Nov 03 21:05:00.198691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.198640 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548"} err="failed to get container status \"15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548\": rpc error: code = NotFound desc = could not find container \"15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548\": container with ID starting with 15c03e0ca281546352d4577b1d0e9842c770388ca7dbc71fe3428356a667d548 not found: ID does not exist" Nov 03 21:05:00.378493 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.378457 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2"] Nov 03 21:05:00.426555 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.426483 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b6d800-openshift-default-67cfd6b4bf-tlpg2"] Nov 03 21:05:00.902566 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:00.902532 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" path="/var/lib/kubelet/pods/ddbea0aa-e253-4668-bdb9-faa486867e9c/volumes" Nov 03 21:05:12.177925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:12.177890 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l"] Nov 03 21:05:12.178502 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:12.178123 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" containerID="cri-o://b142b6af454f50dc2b3ae279777ffc2b4ffd70e02ec4aef59d3e56c40d547c8f" gracePeriod=30 Nov 03 21:05:12.224715 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:12.224680 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq"] Nov 03 21:05:12.224925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:12.224882 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" containerID="cri-o://674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05" gracePeriod=30 Nov 03 21:05:17.335639 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.335603 4227 generic.go:356] "Generic (PLEG): container finished" podID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerID="b142b6af454f50dc2b3ae279777ffc2b4ffd70e02ec4aef59d3e56c40d547c8f" exitCode=0 Nov 03 21:05:17.335981 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.335673 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" event={"ID":"50513e57-b41d-4005-bbfd-956125b1b7dd","Type":"ContainerDied","Data":"b142b6af454f50dc2b3ae279777ffc2b4ffd70e02ec4aef59d3e56c40d547c8f"} Nov 03 21:05:17.520587 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.520561 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 21:05:17.550744 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.550720 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 21:05:17.644900 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.644864 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-credential-socket\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.645059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.644916 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822d773b-ea97-47e6-99d6-9b6b884ea61a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822d773b-ea97-47e6-99d6-9b6b884ea61a-istiod-ca-cert\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.645059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.644940 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-podinfo\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.645059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.644960 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-data\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.645059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.644979 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-data\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.645059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.644996 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-credential-socket\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.645059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645024 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-socket\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.645059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645049 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-kube-api-access-zxdsp\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-kube-api-access-zxdsp\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645079 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-token\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-token\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645187 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-token\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-token\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645240 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-credential-socket" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645279 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50513e57-b41d-4005-bbfd-956125b1b7dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50513e57-b41d-4005-bbfd-956125b1b7dd-istiod-ca-cert\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645313 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-credential-socket" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645353 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-socket" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645383 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-socket\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645392 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-data" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645417 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-envoy\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645429 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822d773b-ea97-47e6-99d6-9b6b884ea61a-istiod-ca-cert" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:17.645501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645458 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-kube-api-access-82xh5\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-kube-api-access-82xh5\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.646030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645506 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-certs\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.646030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645549 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-podinfo\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.646030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645592 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-socket" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.646030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645621 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-envoy\") pod \"50513e57-b41d-4005-bbfd-956125b1b7dd\" (UID: \"50513e57-b41d-4005-bbfd-956125b1b7dd\") " Nov 03 21:05:17.646030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645639 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50513e57-b41d-4005-bbfd-956125b1b7dd-istiod-ca-cert" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:17.646030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.645696 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-certs\") pod \"822d773b-ea97-47e6-99d6-9b6b884ea61a\" (UID: \"822d773b-ea97-47e6-99d6-9b6b884ea61a\") " Nov 03 21:05:17.646030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646022 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-data" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.646400 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646209 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-certs" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.646473 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646452 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646478 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646493 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822d773b-ea97-47e6-99d6-9b6b884ea61a-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646507 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646503 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-certs" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.646532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646521 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646535 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646548 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646562 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50513e57-b41d-4005-bbfd-956125b1b7dd-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.646795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.646576 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.647823 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.647795 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-token" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:17.647935 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.647906 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-podinfo" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:17.648094 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.648061 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-kube-api-access-zxdsp" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "kube-api-access-zxdsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:17.648621 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.648591 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-envoy" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.648800 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.648779 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-token" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:17.649143 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.649118 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-envoy" pod "50513e57-b41d-4005-bbfd-956125b1b7dd" (UID: "50513e57-b41d-4005-bbfd-956125b1b7dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:17.649543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.649526 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-podinfo" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:17.649706 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.649688 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-kube-api-access-82xh5" pod "822d773b-ea97-47e6-99d6-9b6b884ea61a" (UID: "822d773b-ea97-47e6-99d6-9b6b884ea61a"). InnerVolumeSpecName "kube-api-access-82xh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747412 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747439 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxdsp\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-kube-api-access-zxdsp\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747450 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747459 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747469 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747477 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xh5\" (UniqueName: \"kubernetes.io/projected/822d773b-ea97-47e6-99d6-9b6b884ea61a-kube-api-access-82xh5\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747487 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747496 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822d773b-ea97-47e6-99d6-9b6b884ea61a-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:17.747869 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:17.747505 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50513e57-b41d-4005-bbfd-956125b1b7dd-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:18.343072 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.343036 4227 generic.go:356] "Generic (PLEG): container finished" podID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerID="674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05" exitCode=0 Nov 03 21:05:18.343626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.343093 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" Nov 03 21:05:18.343626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.343116 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" event={"ID":"822d773b-ea97-47e6-99d6-9b6b884ea61a","Type":"ContainerDied","Data":"674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05"} Nov 03 21:05:18.343626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.343223 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq" event={"ID":"822d773b-ea97-47e6-99d6-9b6b884ea61a","Type":"ContainerDied","Data":"3b2fd15443f8074831d1212da67c233b5ed4417538b99e951a022b44bd1c479e"} Nov 03 21:05:18.343626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.343249 4227 scope.go:122] "RemoveContainer" containerID="674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05" Nov 03 21:05:18.345055 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.345030 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" Nov 03 21:05:18.345153 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.345079 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l" event={"ID":"50513e57-b41d-4005-bbfd-956125b1b7dd","Type":"ContainerDied","Data":"b7501c4bbe3e9870ab0c7b5a61c159178b8dbad9be88fc37d2848fc3751318f0"} Nov 03 21:05:18.355091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.355071 4227 scope.go:122] "RemoveContainer" containerID="674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05" Nov 03 21:05:18.355470 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:18.355442 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05\": container with ID starting with 674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05 not found: ID does not exist" containerID="674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05" Nov 03 21:05:18.355562 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.355479 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05"} err="failed to get container status \"674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05\": rpc error: code = NotFound desc = could not find container \"674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05\": container with ID starting with 674d6b430e246c595f93bf878584aad377ea2fa87ea8d029f25422269a52ae05 not found: ID does not exist" Nov 03 21:05:18.355562 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.355519 4227 scope.go:122] "RemoveContainer" containerID="b142b6af454f50dc2b3ae279777ffc2b4ffd70e02ec4aef59d3e56c40d547c8f" Nov 03 21:05:18.500475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.500374 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq"] Nov 03 21:05:18.546893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.546855 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-98pkq"] Nov 03 21:05:18.606963 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.606925 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l"] Nov 03 21:05:18.610902 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.610866 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-9522l"] Nov 03 21:05:18.902076 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.902044 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" path="/var/lib/kubelet/pods/50513e57-b41d-4005-bbfd-956125b1b7dd/volumes" Nov 03 21:05:18.902610 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:18.902593 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" path="/var/lib/kubelet/pods/822d773b-ea97-47e6-99d6-9b6b884ea61a/volumes" Nov 03 21:05:21.771184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:21.767566 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght"] Nov 03 21:05:21.771184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:21.767855 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" containerID="cri-o://5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74" gracePeriod=30 Nov 03 21:05:26.102993 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:26.102949 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:05:27.077958 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.077933 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:05:27.173670 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.173642 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66991d08-963b-470c-836c-ee803c475c1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66991d08-963b-470c-836c-ee803c475c1f-istiod-ca-cert\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.173724 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-envoy\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.173777 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-data\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.173809 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-credential-socket\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.173853 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-socket\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.173918 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-kube-api-access-ftgtc\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-kube-api-access-ftgtc\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174063 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-certs\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174067 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66991d08-963b-470c-836c-ee803c475c1f-istiod-ca-cert" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:27.174093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174082 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-data" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:27.174522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174099 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-credential-socket" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:27.174522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174215 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66991d08-963b-470c-836c-ee803c475c1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66991d08-963b-470c-836c-ee803c475c1f-istio-podinfo\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174227 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-socket" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:27.174522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174264 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-istio-token\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-istio-token\") pod \"66991d08-963b-470c-836c-ee803c475c1f\" (UID: \"66991d08-963b-470c-836c-ee803c475c1f\") " Nov 03 21:05:27.174522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.174396 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-certs" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:27.175427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.175338 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.175427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.175366 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.175427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.175382 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.175427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.175396 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.175427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.175410 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66991d08-963b-470c-836c-ee803c475c1f-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.176671 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.176642 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-kube-api-access-ftgtc" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "kube-api-access-ftgtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:27.177263 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.177235 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66991d08-963b-470c-836c-ee803c475c1f-istio-podinfo" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:27.177406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.177341 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-istio-token" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:27.177463 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.177420 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-envoy" pod "66991d08-963b-470c-836c-ee803c475c1f" (UID: "66991d08-963b-470c-836c-ee803c475c1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:27.277216 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.277176 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66991d08-963b-470c-836c-ee803c475c1f-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.277216 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.277214 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.277416 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.277229 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66991d08-963b-470c-836c-ee803c475c1f-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.277416 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.277244 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgtc\" (UniqueName: \"kubernetes.io/projected/66991d08-963b-470c-836c-ee803c475c1f-kube-api-access-ftgtc\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:27.477877 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.477828 4227 generic.go:356] "Generic (PLEG): container finished" podID="66991d08-963b-470c-836c-ee803c475c1f" containerID="5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74" exitCode=0 Nov 03 21:05:27.478052 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.477888 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" Nov 03 21:05:27.478052 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.477916 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" event={"ID":"66991d08-963b-470c-836c-ee803c475c1f","Type":"ContainerDied","Data":"5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74"} Nov 03 21:05:27.478052 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.477987 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght" event={"ID":"66991d08-963b-470c-836c-ee803c475c1f","Type":"ContainerDied","Data":"1d63a4b6a4b32e409f6d9552c302a32da73ed3265465257b244d222c140706e3"} Nov 03 21:05:27.478052 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.478019 4227 scope.go:122] "RemoveContainer" containerID="5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74" Nov 03 21:05:27.490218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.490193 4227 scope.go:122] "RemoveContainer" containerID="449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c" Nov 03 21:05:27.499141 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.499115 4227 scope.go:122] "RemoveContainer" containerID="5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74" Nov 03 21:05:27.499454 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:27.499433 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74\": container with ID starting with 5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74 not found: ID does not exist" containerID="5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74" Nov 03 21:05:27.499517 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.499463 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74"} err="failed to get container status \"5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74\": rpc error: code = NotFound desc = could not find container \"5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74\": container with ID starting with 5f26652d2fef1676b324ed23c29a26be5c4a602911663b92327025e14ef44f74 not found: ID does not exist" Nov 03 21:05:27.499517 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.499484 4227 scope.go:122] "RemoveContainer" containerID="449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c" Nov 03 21:05:27.499763 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:27.499746 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c\": container with ID starting with 449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c not found: ID does not exist" containerID="449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c" Nov 03 21:05:27.499826 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.499773 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c"} err="failed to get container status \"449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c\": rpc error: code = NotFound desc = could not find container \"449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c\": container with ID starting with 449d7619497d5984740442afa317fa8053ea4f22ae8b921ddbe5469a6bb3932c not found: ID does not exist" Nov 03 21:05:27.632819 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.632781 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght"] Nov 03 21:05:27.637743 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:27.637718 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e6aaa6-openshift-default-7fc4b9b6f-dfght"] Nov 03 21:05:28.900993 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:28.900957 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66991d08-963b-470c-836c-ee803c475c1f" path="/var/lib/kubelet/pods/66991d08-963b-470c-836c-ee803c475c1f/volumes" Nov 03 21:05:30.498778 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.498740 4227 generic.go:356] "Generic (PLEG): container finished" podID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerID="7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5" exitCode=0 Nov 03 21:05:30.499304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.498813 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" event={"ID":"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7","Type":"ContainerDied","Data":"7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5"} Nov 03 21:05:30.499379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.499356 4227 scope.go:122] "RemoveContainer" containerID="7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5" Nov 03 21:05:30.501022 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.500990 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:30.501110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.501062 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:30.501110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.501096 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:30.706466 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.706381 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt"] Nov 03 21:05:30.706685 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:30.706658 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" podUID="c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" containerName="maas-api" containerID="cri-o://cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb" gracePeriod=30 Nov 03 21:05:31.329412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.329382 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:05:31.386181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.383569 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-openshift-service-ca\") pod \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " Nov 03 21:05:31.386181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.383623 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-kube-api-access-zqtw8\" (UniqueName: \"kubernetes.io/projected/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-kube-api-access-zqtw8\") pod \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " Nov 03 21:05:31.386181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.383687 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-maas-api-tls\") pod \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " Nov 03 21:05:31.386181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.383843 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-metrics-certs\") pod \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\" (UID: \"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3\") " Nov 03 21:05:31.386181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.385287 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-openshift-service-ca" pod "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" (UID: "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:31.386181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.386067 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:31.392196 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.389629 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-maas-api-tls" pod "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" (UID: "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:31.392196 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.389950 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-kube-api-access-zqtw8" pod "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" (UID: "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3"). InnerVolumeSpecName "kube-api-access-zqtw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:31.393509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.393484 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-metrics-certs" pod "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" (UID: "c1eafa0d-eb9f-4c11-9ab5-56159d3975a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:31.487407 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.487361 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:31.487407 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.487403 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtw8\" (UniqueName: \"kubernetes.io/projected/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-kube-api-access-zqtw8\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:31.487674 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.487421 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:31.508818 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.508785 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" event={"ID":"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7","Type":"ContainerStarted","Data":"6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2"} Nov 03 21:05:31.510323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.510300 4227 generic.go:356] "Generic (PLEG): container finished" podID="c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" containerID="cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb" exitCode=0 Nov 03 21:05:31.510475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.510348 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" Nov 03 21:05:31.510475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.510365 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" event={"ID":"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3","Type":"ContainerDied","Data":"cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb"} Nov 03 21:05:31.510475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.510399 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt" event={"ID":"c1eafa0d-eb9f-4c11-9ab5-56159d3975a3","Type":"ContainerDied","Data":"6139dffa865b9f63bd950cbe3e364253896662d5b17b7cbf2536664ad2c3e052"} Nov 03 21:05:31.510475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.510421 4227 scope.go:122] "RemoveContainer" containerID="cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb" Nov 03 21:05:31.520839 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.520820 4227 scope.go:122] "RemoveContainer" containerID="cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb" Nov 03 21:05:31.521130 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:31.521109 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb\": container with ID starting with cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb not found: ID does not exist" containerID="cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb" Nov 03 21:05:31.521214 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.521140 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb"} err="failed to get container status \"cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb\": rpc error: code = NotFound desc = could not find container \"cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb\": container with ID starting with cb56edd64792071e8a5595515646bad9a8f5873a0c253b0531f59eb168cd56cb not found: ID does not exist" Nov 03 21:05:31.757232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.755242 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt"] Nov 03 21:05:31.779183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:31.770932 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-20d356-977fd8d68-kj2kt"] Nov 03 21:05:32.474706 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:32.474662 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:05:32.474919 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:32.474815 4227 patch_prober.go:29] interesting pod/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:05:32.474919 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:32.474873 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 21:05:33.011791 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:33.011746 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" path="/var/lib/kubelet/pods/c1eafa0d-eb9f-4c11-9ab5-56159d3975a3/volumes" Nov 03 21:05:33.477181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:33.474979 4227 patch_prober.go:29] interesting pod/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:05:33.477181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:33.475041 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 21:05:34.494242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:34.478061 4227 patch_prober.go:29] interesting pod/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:05:34.494242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:34.478132 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 21:05:35.475184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:35.474588 4227 patch_prober.go:29] interesting pod/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:05:35.475184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:35.474656 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 21:05:36.475439 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:36.475400 4227 patch_prober.go:29] interesting pod/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 21:05:36.475874 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:36.475466 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 21:05:37.367755 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.367713 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk"] Nov 03 21:05:37.368019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.367985 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" podUID="daad49ee-ca87-4d5a-aebd-528d7e672374" containerName="maas-api" containerID="cri-o://62a6351fcfbdb9c82d078216ee15f86306f1022f46797bacaf69fc4cec40a839" gracePeriod=30 Nov 03 21:05:37.421056 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.421007 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" podUID="daad49ee-ca87-4d5a-aebd-528d7e672374" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": dial tcp 10.134.0.84:8443: connect: connection refused" Nov 03 21:05:37.482221 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.481959 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:05:37.482746 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.482638 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:05:37.483706 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.483685 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:05:37.685587 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.685537 4227 generic.go:356] "Generic (PLEG): container finished" podID="daad49ee-ca87-4d5a-aebd-528d7e672374" containerID="62a6351fcfbdb9c82d078216ee15f86306f1022f46797bacaf69fc4cec40a839" exitCode=0 Nov 03 21:05:37.687036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.686800 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" event={"ID":"daad49ee-ca87-4d5a-aebd-528d7e672374","Type":"ContainerDied","Data":"62a6351fcfbdb9c82d078216ee15f86306f1022f46797bacaf69fc4cec40a839"} Nov 03 21:05:37.719381 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.719349 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 21:05:37.871570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.871531 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls\") pod \"daad49ee-ca87-4d5a-aebd-528d7e672374\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " Nov 03 21:05:37.871750 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.871683 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daad49ee-ca87-4d5a-aebd-528d7e672374-kube-api-access-xp4vz\" (UniqueName: \"kubernetes.io/projected/daad49ee-ca87-4d5a-aebd-528d7e672374-kube-api-access-xp4vz\") pod \"daad49ee-ca87-4d5a-aebd-528d7e672374\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " Nov 03 21:05:37.871750 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.871724 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-metrics-certs\") pod \"daad49ee-ca87-4d5a-aebd-528d7e672374\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " Nov 03 21:05:37.871864 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.871845 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daad49ee-ca87-4d5a-aebd-528d7e672374-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad49ee-ca87-4d5a-aebd-528d7e672374-openshift-service-ca\") pod \"daad49ee-ca87-4d5a-aebd-528d7e672374\" (UID: \"daad49ee-ca87-4d5a-aebd-528d7e672374\") " Nov 03 21:05:37.872703 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.872677 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daad49ee-ca87-4d5a-aebd-528d7e672374-openshift-service-ca" pod "daad49ee-ca87-4d5a-aebd-528d7e672374" (UID: "daad49ee-ca87-4d5a-aebd-528d7e672374"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:37.877246 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.877216 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daad49ee-ca87-4d5a-aebd-528d7e672374-kube-api-access-xp4vz" pod "daad49ee-ca87-4d5a-aebd-528d7e672374" (UID: "daad49ee-ca87-4d5a-aebd-528d7e672374"). InnerVolumeSpecName "kube-api-access-xp4vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:37.877352 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.877255 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls" pod "daad49ee-ca87-4d5a-aebd-528d7e672374" (UID: "daad49ee-ca87-4d5a-aebd-528d7e672374"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:37.877488 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.877467 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-metrics-certs" pod "daad49ee-ca87-4d5a-aebd-528d7e672374" (UID: "daad49ee-ca87-4d5a-aebd-528d7e672374"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:37.977228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.976149 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:37.977228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.976211 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4vz\" (UniqueName: \"kubernetes.io/projected/daad49ee-ca87-4d5a-aebd-528d7e672374-kube-api-access-xp4vz\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:37.977228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.976230 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad49ee-ca87-4d5a-aebd-528d7e672374-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:37.977228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:37.976244 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad49ee-ca87-4d5a-aebd-528d7e672374-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.693656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:38.693622 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" Nov 03 21:05:38.694186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:38.693654 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk" event={"ID":"daad49ee-ca87-4d5a-aebd-528d7e672374","Type":"ContainerDied","Data":"bb59125c0490bcab3e5596a86eae662fcb757947e06ac1085c24df8ca0866693"} Nov 03 21:05:38.694186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:38.693715 4227 scope.go:122] "RemoveContainer" containerID="62a6351fcfbdb9c82d078216ee15f86306f1022f46797bacaf69fc4cec40a839" Nov 03 21:05:38.849234 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:38.849119 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk"] Nov 03 21:05:38.909305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:38.909270 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-211d68bf-dd9f9d9b4-s2hlk"] Nov 03 21:05:39.890569 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:39.890532 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg"] Nov 03 21:05:39.891043 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:39.890791 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" containerID="cri-o://4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee" gracePeriod=30 Nov 03 21:05:39.918794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:39.918758 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5"] Nov 03 21:05:39.919071 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:39.919021 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" containerID="cri-o://8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652" gracePeriod=30 Nov 03 21:05:40.902207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:40.902175 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daad49ee-ca87-4d5a-aebd-528d7e672374" path="/var/lib/kubelet/pods/daad49ee-ca87-4d5a-aebd-528d7e672374/volumes" Nov 03 21:05:43.073909 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.073871 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh"] Nov 03 21:05:43.075382 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075354 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" containerName="maas-api" Nov 03 21:05:43.075382 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075380 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" containerName="maas-api" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075403 4227 cpu_manager.go:414] "containerMap: removing container" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075411 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075460 4227 cpu_manager.go:414] "containerMap: removing container" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075467 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075482 4227 cpu_manager.go:414] "containerMap: removing container" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075489 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075498 4227 cpu_manager.go:414] "containerMap: removing container" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075506 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075518 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075525 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075539 4227 cpu_manager.go:414] "containerMap: removing container" podUID="daad49ee-ca87-4d5a-aebd-528d7e672374" containerName="maas-api" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075546 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daad49ee-ca87-4d5a-aebd-528d7e672374" containerName="maas-api" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075559 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" Nov 03 21:05:43.075570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075566 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075696 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="50513e57-b41d-4005-bbfd-956125b1b7dd" containerName="istio-proxy" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075716 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="822d773b-ea97-47e6-99d6-9b6b884ea61a" containerName="istio-proxy" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075726 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075736 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="66991d08-963b-470c-836c-ee803c475c1f" containerName="istio-proxy" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075745 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075753 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="daad49ee-ca87-4d5a-aebd-528d7e672374" containerName="maas-api" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075761 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbea0aa-e253-4668-bdb9-faa486867e9c" containerName="istio-proxy" Nov 03 21:05:43.076316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.075771 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1eafa0d-eb9f-4c11-9ab5-56159d3975a3" containerName="maas-api" Nov 03 21:05:43.080124 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.079984 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh"] Nov 03 21:05:43.080269 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.080149 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk"] Nov 03 21:05:43.080269 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.080087 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.084369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.084344 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:43.110583 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.110551 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk"] Nov 03 21:05:43.131614 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.131578 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7c34ff-openshift-default-dockercfg-mxczf\"" Nov 03 21:05:43.132667 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.132642 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c14420-dockercfg-f98sj\"" Nov 03 21:05:43.164181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.153130 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c14420\"" Nov 03 21:05:43.300185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.253483 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-podinfo\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.305726 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-data\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.305804 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c84ab14-c774-41c4-852a-4eb8698c83ea-istiod-ca-cert\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.305867 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcb5l\" (UniqueName: \"kubernetes.io/projected/bd24478d-4d06-4727-a7e9-7b392025bd96-kube-api-access-zcb5l\") pod \"payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.305892 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-credential-socket\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.306038 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-envoy\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.306092 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxms9\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-kube-api-access-bxms9\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.306121 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-certs\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.306231 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-socket\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.306264 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd24478d-4d06-4727-a7e9-7b392025bd96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.306290 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-token\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.308182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.300206 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc"] Nov 03 21:05:43.317969 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.317933 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc"] Nov 03 21:05:43.318345 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.318323 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.407482 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407440 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcb5l\" (UniqueName: \"kubernetes.io/projected/bd24478d-4d06-4727-a7e9-7b392025bd96-kube-api-access-zcb5l\") pod \"payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:43.407482 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407487 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-credential-socket\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.407759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407590 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-envoy\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.407759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407632 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmx4c\" (UniqueName: \"kubernetes.io/projected/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-kube-api-access-xmx4c\") pod \"payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.407759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407661 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxms9\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-kube-api-access-bxms9\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.407759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407686 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-certs\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.407759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407739 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.408019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407783 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-socket\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407820 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd24478d-4d06-4727-a7e9-7b392025bd96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:43.408019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407846 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-token\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407941 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-podinfo\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407992 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-data\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.407999 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-credential-socket\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408351 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.408039 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c84ab14-c774-41c4-852a-4eb8698c83ea-istiod-ca-cert\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408351 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.408314 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-socket\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408566 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.408473 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-certs\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.408652 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.408624 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-data\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.409009 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.408983 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd24478d-4d06-4727-a7e9-7b392025bd96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:43.409239 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.409222 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c84ab14-c774-41c4-852a-4eb8698c83ea-istiod-ca-cert\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.410460 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.410440 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-envoy\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.410549 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.410519 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-podinfo\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.462155 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.462115 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-token\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.462340 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.462234 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcb5l\" (UniqueName: \"kubernetes.io/projected/bd24478d-4d06-4727-a7e9-7b392025bd96-kube-api-access-zcb5l\") pod \"payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:43.462471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.462447 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxms9\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-kube-api-access-bxms9\") pod \"e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.509699 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.509665 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmx4c\" (UniqueName: \"kubernetes.io/projected/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-kube-api-access-xmx4c\") pod \"payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.509942 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.509717 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.510437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.510417 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.589223 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.573901 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmx4c\" (UniqueName: \"kubernetes.io/projected/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-kube-api-access-xmx4c\") pod \"payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.647183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.647129 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:43.703637 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.703445 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:43.711281 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:43.711253 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:44.030738 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.030698 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc"] Nov 03 21:05:44.035332 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:05:44.035304 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7637d2_a8a2_4b8d_81e2_6d6af3f71b8c.slice/crio-6785509ce301199577d4df9345b7a8d3c8514f62a318334a06a5563292e2f590 WatchSource:0}: Error finding container 6785509ce301199577d4df9345b7a8d3c8514f62a318334a06a5563292e2f590: Status 404 returned error can't find the container with id 6785509ce301199577d4df9345b7a8d3c8514f62a318334a06a5563292e2f590 Nov 03 21:05:44.144662 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.144623 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk"] Nov 03 21:05:44.149425 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:05:44.149380 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd24478d_4d06_4727_a7e9_7b392025bd96.slice/crio-a414e7a6a486e7c0510bb88aa7fd7fcc4baee8a4be8e082e2637450192150b21 WatchSource:0}: Error finding container a414e7a6a486e7c0510bb88aa7fd7fcc4baee8a4be8e082e2637450192150b21: Status 404 returned error can't find the container with id a414e7a6a486e7c0510bb88aa7fd7fcc4baee8a4be8e082e2637450192150b21 Nov 03 21:05:44.174577 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.174548 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh"] Nov 03 21:05:44.183586 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:05:44.183559 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c84ab14_c774_41c4_852a_4eb8698c83ea.slice/crio-6bddd68463084c49485c018f439d05273b6c2c54bc28f5310f8dc0f135c10a4a WatchSource:0}: Error finding container 6bddd68463084c49485c018f439d05273b6c2c54bc28f5310f8dc0f135c10a4a: Status 404 returned error can't find the container with id 6bddd68463084c49485c018f439d05273b6c2c54bc28f5310f8dc0f135c10a4a Nov 03 21:05:44.186960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.186917 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:44.187089 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.187064 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:44.187179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.187133 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:44.743860 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.743824 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" event={"ID":"9c84ab14-c774-41c4-852a-4eb8698c83ea","Type":"ContainerStarted","Data":"83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975"} Nov 03 21:05:44.744049 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.743882 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" event={"ID":"9c84ab14-c774-41c4-852a-4eb8698c83ea","Type":"ContainerStarted","Data":"6bddd68463084c49485c018f439d05273b6c2c54bc28f5310f8dc0f135c10a4a"} Nov 03 21:05:44.745464 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.745435 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" event={"ID":"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c","Type":"ContainerStarted","Data":"7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df"} Nov 03 21:05:44.745610 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.745493 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:44.745610 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.745520 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" event={"ID":"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c","Type":"ContainerStarted","Data":"6785509ce301199577d4df9345b7a8d3c8514f62a318334a06a5563292e2f590"} Nov 03 21:05:44.747201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.747139 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" event={"ID":"bd24478d-4d06-4727-a7e9-7b392025bd96","Type":"ContainerStarted","Data":"8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80"} Nov 03 21:05:44.747319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.747225 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" event={"ID":"bd24478d-4d06-4727-a7e9-7b392025bd96","Type":"ContainerStarted","Data":"a414e7a6a486e7c0510bb88aa7fd7fcc4baee8a4be8e082e2637450192150b21"} Nov 03 21:05:44.747319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.747255 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:44.871815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.871750 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podStartSLOduration=2.871732539 podStartE2EDuration="2.871732539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:44.870318752 +0000 UTC m=+2212.107296642" watchObservedRunningTime="2026-11-03 21:05:44.871732539 +0000 UTC m=+2212.108710414" Nov 03 21:05:44.987151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:44.987098 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" podStartSLOduration=1.987082997 podStartE2EDuration="1.987082997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:44.985967205 +0000 UTC m=+2212.222945079" watchObservedRunningTime="2026-11-03 21:05:44.987082997 +0000 UTC m=+2212.224060870" Nov 03 21:05:45.140787 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.096992 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" podStartSLOduration=2.09697077 podStartE2EDuration="2.09697077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:45.092496517 +0000 UTC m=+2212.329474391" watchObservedRunningTime="2026-11-03 21:05:45.09697077 +0000 UTC m=+2212.333948646" Nov 03 21:05:45.333147 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.333112 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:05:45.343166 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343131 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-certs\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343250 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istiod-ca-cert\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343296 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-podinfo\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343429 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343326 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-token\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-token\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343429 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343383 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-credential-socket\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343429 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343393 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-certs" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.343626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343429 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-envoy\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343517 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-socket\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343605 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-kube-api-access-kkcfq\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-kube-api-access-kkcfq\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343634 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-data\") pod \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\" (UID: \"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e\") " Nov 03 21:05:45.343797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343687 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-credential-socket" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.343893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343872 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-socket" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.343945 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.343900 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istiod-ca-cert" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:45.344128 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.344104 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-data" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.344240 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.344219 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.344311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.344255 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.344311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.344266 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.344311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.344275 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.346496 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.346407 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-token" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.346496 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.346427 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-podinfo" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:45.346684 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.346591 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-kube-api-access-kkcfq" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "kube-api-access-kkcfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.347264 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.347230 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-envoy" pod "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" (UID: "64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.448662 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.448546 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkcfq\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-kube-api-access-kkcfq\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.448662 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.448583 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.448662 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.448598 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.448662 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.448611 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.448662 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.448625 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.600383 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.600359 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:05:45.651243 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651206 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-certs\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651249 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-data\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651310 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-envoy\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651384 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-istio-token\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-istio-token\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651424 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-socket\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651478 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08577ba9-52bb-4c8c-8879-63deb9968296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08577ba9-52bb-4c8c-8879-63deb9968296-istiod-ca-cert\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651498 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-certs" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.651576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651540 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-credential-socket\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651564 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-data" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.651576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651571 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08577ba9-52bb-4c8c-8879-63deb9968296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08577ba9-52bb-4c8c-8879-63deb9968296-istio-podinfo\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.651864 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651753 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-socket" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.651924 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651891 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-credential-socket" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.651981 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.651922 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08577ba9-52bb-4c8c-8879-63deb9968296-istiod-ca-cert" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:45.652147 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.652128 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.652319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.652153 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08577ba9-52bb-4c8c-8879-63deb9968296-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.652319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.652199 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.652319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.652212 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.652319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.652225 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.654253 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.654225 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-envoy" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:45.654253 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.654238 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-istio-token" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.654386 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.654253 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08577ba9-52bb-4c8c-8879-63deb9968296-istio-podinfo" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:45.704600 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.704574 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:45.710854 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.710826 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:45.753361 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.753325 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-kube-api-access-sb2ck\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-kube-api-access-sb2ck\") pod \"08577ba9-52bb-4c8c-8879-63deb9968296\" (UID: \"08577ba9-52bb-4c8c-8879-63deb9968296\") " Nov 03 21:05:45.753752 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.753728 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08577ba9-52bb-4c8c-8879-63deb9968296-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.753752 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.753751 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08577ba9-52bb-4c8c-8879-63deb9968296-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.753960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.753763 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:45.756146 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.756121 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-kube-api-access-sb2ck" pod "08577ba9-52bb-4c8c-8879-63deb9968296" (UID: "08577ba9-52bb-4c8c-8879-63deb9968296"). InnerVolumeSpecName "kube-api-access-sb2ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:45.758102 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.758066 4227 generic.go:356] "Generic (PLEG): container finished" podID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerID="8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652" exitCode=0 Nov 03 21:05:45.758222 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.758133 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" Nov 03 21:05:45.758325 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.758304 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" event={"ID":"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e","Type":"ContainerDied","Data":"8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652"} Nov 03 21:05:45.758400 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.758386 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5" event={"ID":"64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e","Type":"ContainerDied","Data":"a1e9640cd381b54869a3a9495324606126c010d5863be53f8b34b177bc5aa422"} Nov 03 21:05:45.758457 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.758418 4227 scope.go:122] "RemoveContainer" containerID="8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652" Nov 03 21:05:45.761410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.761374 4227 generic.go:356] "Generic (PLEG): container finished" podID="08577ba9-52bb-4c8c-8879-63deb9968296" containerID="4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee" exitCode=0 Nov 03 21:05:45.762494 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.762465 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" Nov 03 21:05:45.766052 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.766029 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" event={"ID":"08577ba9-52bb-4c8c-8879-63deb9968296","Type":"ContainerDied","Data":"4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee"} Nov 03 21:05:45.766154 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.766074 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg" event={"ID":"08577ba9-52bb-4c8c-8879-63deb9968296","Type":"ContainerDied","Data":"788d3949bcf5184dd935efd1b72b70774d237207c53efdea753b90abaa9061bd"} Nov 03 21:05:45.766154 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.766103 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:45.767272 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.767254 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:05:45.771519 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.771503 4227 scope.go:122] "RemoveContainer" containerID="8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652" Nov 03 21:05:45.771826 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:45.771804 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652\": container with ID starting with 8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652 not found: ID does not exist" containerID="8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652" Nov 03 21:05:45.771911 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.771839 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652"} err="failed to get container status \"8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652\": rpc error: code = NotFound desc = could not find container \"8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652\": container with ID starting with 8c7939db780d6d6b531bdaaf2efe9d62f1874cb53d948b59c757e99481412652 not found: ID does not exist" Nov 03 21:05:45.771911 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.771862 4227 scope.go:122] "RemoveContainer" containerID="4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee" Nov 03 21:05:45.788496 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.788469 4227 scope.go:122] "RemoveContainer" containerID="7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53" Nov 03 21:05:45.798784 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.798760 4227 scope.go:122] "RemoveContainer" containerID="4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee" Nov 03 21:05:45.799041 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:45.799023 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee\": container with ID starting with 4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee not found: ID does not exist" containerID="4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee" Nov 03 21:05:45.799091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.799049 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee"} err="failed to get container status \"4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee\": rpc error: code = NotFound desc = could not find container \"4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee\": container with ID starting with 4cf366ab25f9144dbb1ba798501ba99498e23b6ebeadf82154b805b65d7ce5ee not found: ID does not exist" Nov 03 21:05:45.799091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.799067 4227 scope.go:122] "RemoveContainer" containerID="7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53" Nov 03 21:05:45.799574 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:45.799311 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53\": container with ID starting with 7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53 not found: ID does not exist" containerID="7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53" Nov 03 21:05:45.799574 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.799347 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53"} err="failed to get container status \"7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53\": rpc error: code = NotFound desc = could not find container \"7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53\": container with ID starting with 7a367e2f591ea80718af117c5d2ea24b601c839d9ec80c8f3ab52766f4a18e53 not found: ID does not exist" Nov 03 21:05:45.855321 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:45.855233 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2ck\" (UniqueName: \"kubernetes.io/projected/08577ba9-52bb-4c8c-8879-63deb9968296-kube-api-access-sb2ck\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:46.034183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:46.034126 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5"] Nov 03 21:05:46.042885 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:46.042857 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-v5kw5"] Nov 03 21:05:46.150136 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:46.150105 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg"] Nov 03 21:05:46.150493 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:46.150467 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-8gqmg"] Nov 03 21:05:46.902831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:46.902803 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" path="/var/lib/kubelet/pods/08577ba9-52bb-4c8c-8879-63deb9968296/volumes" Nov 03 21:05:46.903332 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:46.903317 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" path="/var/lib/kubelet/pods/64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e/volumes" Nov 03 21:05:55.769044 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:55.769012 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:05:55.769504 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:05:55.769354 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:05:56.081787 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:05:56.081703 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:06:12.634350 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.634313 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw"] Nov 03 21:06:12.634927 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.634910 4227 cpu_manager.go:414] "containerMap: removing container" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" Nov 03 21:06:12.634974 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.634928 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" Nov 03 21:06:12.634974 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.634947 4227 cpu_manager.go:414] "containerMap: removing container" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" Nov 03 21:06:12.634974 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.634953 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" Nov 03 21:06:12.634974 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.634963 4227 cpu_manager.go:414] "containerMap: removing container" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" Nov 03 21:06:12.634974 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.634968 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" Nov 03 21:06:12.635138 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.635050 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" Nov 03 21:06:12.635138 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.635060 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="08577ba9-52bb-4c8c-8879-63deb9968296" containerName="istio-proxy" Nov 03 21:06:12.635138 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.635066 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ffea4b-bef0-4e2c-b7e0-b83e84bd9b1e" containerName="istio-proxy" Nov 03 21:06:12.638889 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.638870 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.680727 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.680697 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm"] Nov 03 21:06:12.684196 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.684179 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.710848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.710818 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.710961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.710857 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.710961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.710938 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.711070 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.710984 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.711140 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.711114 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.711250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.711231 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.711316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.711285 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.711421 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.711407 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.711461 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.711434 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4lz\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-kube-api-access-2f4lz\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.728569 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.728544 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw"] Nov 03 21:06:12.779506 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.779477 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm"] Nov 03 21:06:12.812678 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.812644 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.812845 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.812721 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.812845 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.812748 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.812845 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.812789 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7948db-b805-48a7-98b2-c3c27c227963-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.812845 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.812829 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813076 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.812916 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813076 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.812976 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813222 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813111 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813222 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813149 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813222 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813196 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7948db-b805-48a7-98b2-c3c27c227963-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813235 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4lz\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-kube-api-access-2f4lz\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813241 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813280 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813311 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813348 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813379 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813694 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813413 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jsf\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-kube-api-access-r4jsf\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.813694 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813449 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813694 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813448 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813694 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813632 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813694 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813648 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.813962 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813745 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.814028 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.813993 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.815521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.815491 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.815962 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.815938 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.867266 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.867222 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.867415 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.867232 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4lz\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-kube-api-access-2f4lz\") pod \"maas-default-gateway-openshift-default-696f47c6c-jsdpw\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.914752 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.914712 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7948db-b805-48a7-98b2-c3c27c227963-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.914931 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.914770 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.914931 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.914791 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.914931 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.914828 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.914931 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.914907 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915199 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.914939 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7948db-b805-48a7-98b2-c3c27c227963-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915199 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.914991 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915199 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.915022 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915199 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.915041 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jsf\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-kube-api-access-r4jsf\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915424 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.915203 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.915500 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7948db-b805-48a7-98b2-c3c27c227963-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915641 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.915555 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915641 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.915579 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.915715 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.915685 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.917492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.917462 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.917782 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.917764 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7948db-b805-48a7-98b2-c3c27c227963-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.951302 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.951274 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:12.968338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.968312 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.968469 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.968453 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jsf\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-kube-api-access-r4jsf\") pod \"maas-default-gateway-openshift-default-696f47c6c-mlsmm\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:12.997121 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:12.997086 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:13.379343 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.379264 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw"] Nov 03 21:06:13.427303 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:06:13.427274 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b4c192_f4f6_4491_8853_00bc7b865b5d.slice/crio-5a42e35a8bfb891181c0d0f04b50d77b8389205eda033132deec1ff0885f357d WatchSource:0}: Error finding container 5a42e35a8bfb891181c0d0f04b50d77b8389205eda033132deec1ff0885f357d: Status 404 returned error can't find the container with id 5a42e35a8bfb891181c0d0f04b50d77b8389205eda033132deec1ff0885f357d Nov 03 21:06:13.429935 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.429846 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:06:13.430104 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.429960 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:06:13.430104 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.430009 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:06:13.434913 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.434887 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm"] Nov 03 21:06:13.440398 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:06:13.440375 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7948db_b805_48a7_98b2_c3c27c227963.slice/crio-97f8acce098e4b92e02ccbb26e4b2c667d4e6a4b3b6bde461bac2ea610918330 WatchSource:0}: Error finding container 97f8acce098e4b92e02ccbb26e4b2c667d4e6a4b3b6bde461bac2ea610918330: Status 404 returned error can't find the container with id 97f8acce098e4b92e02ccbb26e4b2c667d4e6a4b3b6bde461bac2ea610918330 Nov 03 21:06:13.443301 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.443274 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:06:13.443380 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.443333 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:06:13.443380 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.443365 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:06:13.959078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.959032 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" event={"ID":"c4b4c192-f4f6-4491-8853-00bc7b865b5d","Type":"ContainerStarted","Data":"b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d"} Nov 03 21:06:13.959702 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.959661 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" event={"ID":"c4b4c192-f4f6-4491-8853-00bc7b865b5d","Type":"ContainerStarted","Data":"5a42e35a8bfb891181c0d0f04b50d77b8389205eda033132deec1ff0885f357d"} Nov 03 21:06:13.986191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.979469 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" event={"ID":"5b7948db-b805-48a7-98b2-c3c27c227963","Type":"ContainerStarted","Data":"25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e"} Nov 03 21:06:13.986191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.979589 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" event={"ID":"5b7948db-b805-48a7-98b2-c3c27c227963","Type":"ContainerStarted","Data":"97f8acce098e4b92e02ccbb26e4b2c667d4e6a4b3b6bde461bac2ea610918330"} Nov 03 21:06:13.997204 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:13.997149 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:14.070587 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:14.070520 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podStartSLOduration=2.070500729 podStartE2EDuration="2.070500729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:14.066748099 +0000 UTC m=+2241.303725996" watchObservedRunningTime="2026-11-03 21:06:14.070500729 +0000 UTC m=+2241.307478605" Nov 03 21:06:14.181186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:14.180558 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podStartSLOduration=2.180536764 podStartE2EDuration="2.180536764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:14.177678284 +0000 UTC m=+2241.414656196" watchObservedRunningTime="2026-11-03 21:06:14.180536764 +0000 UTC m=+2241.417514639" Nov 03 21:06:14.967205 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:14.966551 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:14.976194 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:14.972473 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:06:14.976194 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:14.972536 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:06:15.001187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:15.001078 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:06:15.001187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:15.001151 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:06:15.955707 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:15.955671 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:06:15.956029 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:15.955979 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:06:16.001186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:15.999236 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:06:16.001186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:15.999408 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:06:16.958185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:16.955586 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:06:16.958185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:16.955651 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:06:16.997615 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:16.997564 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:06:16.997815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:16.997647 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:06:17.955187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:17.954582 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:06:17.955187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:17.954662 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:06:18.003186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:17.997637 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:06:18.003186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:17.997713 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:06:18.956012 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:18.955975 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:06:18.976855 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:18.976377 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:06:18.998563 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:18.998525 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:06:18.998858 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:18.998827 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:06:19.436960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:19.436920 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6"] Nov 03 21:06:19.542612 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:19.542574 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6"] Nov 03 21:06:19.951696 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:19.951655 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:06:19.951875 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:19.951730 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:06:19.997857 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:19.997816 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:06:19.998257 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:19.997887 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:06:20.965182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:20.954805 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:06:20.965182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:20.954891 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:06:21.003054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:21.003022 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:06:21.003850 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:21.003813 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:06:22.879027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:22.878996 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:22.881118 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:22.880995 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:22.884807 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:22.884774 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:06:22.920990 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:22.920948 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:22.921600 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:22.921574 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:22.923878 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:22.923850 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:06:24.437343 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.437301 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" podUID="6ec6f49d-4114-4945-929c-f5a6b4c70679" containerName="payload-processing" containerID="cri-o://8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1" gracePeriod=25 Nov 03 21:06:24.543151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.543111 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" podUID="d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" containerName="payload-pre-processing" containerID="cri-o://4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d" gracePeriod=25 Nov 03 21:06:24.731052 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.731024 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:06:24.811673 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.811652 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:06:24.846414 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.846378 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec6f49d-4114-4945-929c-f5a6b4c70679-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec6f49d-4114-4945-929c-f5a6b4c70679-plugins-config-volume\") pod \"6ec6f49d-4114-4945-929c-f5a6b4c70679\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " Nov 03 21:06:24.846693 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.846672 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec6f49d-4114-4945-929c-f5a6b4c70679-kube-api-access-qf4sz\" (UniqueName: \"kubernetes.io/projected/6ec6f49d-4114-4945-929c-f5a6b4c70679-kube-api-access-qf4sz\") pod \"6ec6f49d-4114-4945-929c-f5a6b4c70679\" (UID: \"6ec6f49d-4114-4945-929c-f5a6b4c70679\") " Nov 03 21:06:24.849685 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.847484 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec6f49d-4114-4945-929c-f5a6b4c70679-plugins-config-volume" pod "6ec6f49d-4114-4945-929c-f5a6b4c70679" (UID: "6ec6f49d-4114-4945-929c-f5a6b4c70679"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:24.850473 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.850057 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec6f49d-4114-4945-929c-f5a6b4c70679-kube-api-access-qf4sz" pod "6ec6f49d-4114-4945-929c-f5a6b4c70679" (UID: "6ec6f49d-4114-4945-929c-f5a6b4c70679"). InnerVolumeSpecName "kube-api-access-qf4sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:24.948432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.948344 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-kube-api-access-wvn4t\" (UniqueName: \"kubernetes.io/projected/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-kube-api-access-wvn4t\") pod \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " Nov 03 21:06:24.948578 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.948486 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-plugins-config-volume\") pod \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\" (UID: \"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d\") " Nov 03 21:06:24.948932 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.948900 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-plugins-config-volume" pod "d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" (UID: "d7f6d2d1-a249-418c-9b6b-832ad3f7f67d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:24.949066 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.949026 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4sz\" (UniqueName: \"kubernetes.io/projected/6ec6f49d-4114-4945-929c-f5a6b4c70679-kube-api-access-qf4sz\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.949066 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.949046 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec6f49d-4114-4945-929c-f5a6b4c70679-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.949066 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.949060 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.950848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:24.950820 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-kube-api-access-wvn4t" pod "d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" (UID: "d7f6d2d1-a249-418c-9b6b-832ad3f7f67d"). InnerVolumeSpecName "kube-api-access-wvn4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:25.050479 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.050440 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvn4t\" (UniqueName: \"kubernetes.io/projected/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d-kube-api-access-wvn4t\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:25.136480 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.136445 4227 generic.go:356] "Generic (PLEG): container finished" podID="6ec6f49d-4114-4945-929c-f5a6b4c70679" containerID="8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1" exitCode=0 Nov 03 21:06:25.136656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.136492 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" Nov 03 21:06:25.136656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.136534 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" event={"ID":"6ec6f49d-4114-4945-929c-f5a6b4c70679","Type":"ContainerDied","Data":"8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1"} Nov 03 21:06:25.136656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.136592 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6" event={"ID":"6ec6f49d-4114-4945-929c-f5a6b4c70679","Type":"ContainerDied","Data":"fdb301cf9d81b5a9328c7635fe2b08509fc308d2e102a18b07afb3cc87b25420"} Nov 03 21:06:25.136656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.136625 4227 scope.go:122] "RemoveContainer" containerID="8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1" Nov 03 21:06:25.137979 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.137951 4227 generic.go:356] "Generic (PLEG): container finished" podID="d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" containerID="4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d" exitCode=0 Nov 03 21:06:25.138117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.137984 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" Nov 03 21:06:25.138117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.138044 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" event={"ID":"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d","Type":"ContainerDied","Data":"4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d"} Nov 03 21:06:25.138117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.138088 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6" event={"ID":"d7f6d2d1-a249-418c-9b6b-832ad3f7f67d","Type":"ContainerDied","Data":"70760ba977f6a193e3e8918f61cdaf888d023f76f80a1691600a9293fedeee3b"} Nov 03 21:06:25.146354 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.146335 4227 scope.go:122] "RemoveContainer" containerID="8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1" Nov 03 21:06:25.146637 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:06:25.146615 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1\": container with ID starting with 8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1 not found: ID does not exist" containerID="8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1" Nov 03 21:06:25.146753 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.146645 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1"} err="failed to get container status \"8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1\": rpc error: code = NotFound desc = could not find container \"8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1\": container with ID starting with 8ef716a6fb9c6118ca68fd8fad2fa7eb0a6a70efe7d19ee88cb07b3de84bcae1 not found: ID does not exist" Nov 03 21:06:25.146753 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.146661 4227 scope.go:122] "RemoveContainer" containerID="4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d" Nov 03 21:06:25.157241 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.157224 4227 scope.go:122] "RemoveContainer" containerID="4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d" Nov 03 21:06:25.157558 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:06:25.157529 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d\": container with ID starting with 4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d not found: ID does not exist" containerID="4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d" Nov 03 21:06:25.157659 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.157565 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d"} err="failed to get container status \"4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d\": rpc error: code = NotFound desc = could not find container \"4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d\": container with ID starting with 4b16dce234bab3855f607426e414a945c2ba9425ecd2a719c8ce02effb15533d not found: ID does not exist" Nov 03 21:06:25.289899 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.289807 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6"] Nov 03 21:06:25.294638 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.294610 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-46ca0b-5f5fbc545f-5fjn6"] Nov 03 21:06:25.394203 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.394155 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6"] Nov 03 21:06:25.398112 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:25.398085 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-46ca0b-6b64fd7875-6cbk6"] Nov 03 21:06:26.083493 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:06:26.083455 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:06:26.900657 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:26.900627 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec6f49d-4114-4945-929c-f5a6b4c70679" path="/var/lib/kubelet/pods/6ec6f49d-4114-4945-929c-f5a6b4c70679/volumes" Nov 03 21:06:26.901074 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:26.901059 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" path="/var/lib/kubelet/pods/d7f6d2d1-a249-418c-9b6b-832ad3f7f67d/volumes" Nov 03 21:06:37.825357 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:37.801112 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:37.919183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:37.910261 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:39.496693 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.496658 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch"] Nov 03 21:06:39.497147 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.496851 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" podUID="c2507a90-02cd-4107-9df9-6fba6a9240bd" containerName="maas-api" containerID="cri-o://80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255" gracePeriod=30 Nov 03 21:06:39.893211 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.891317 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:06:39.943376 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.941676 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2507a90-02cd-4107-9df9-6fba6a9240bd-kube-api-access-j4f7l\" (UniqueName: \"kubernetes.io/projected/c2507a90-02cd-4107-9df9-6fba6a9240bd-kube-api-access-j4f7l\") pod \"c2507a90-02cd-4107-9df9-6fba6a9240bd\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " Nov 03 21:06:39.943376 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.941724 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2507a90-02cd-4107-9df9-6fba6a9240bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2507a90-02cd-4107-9df9-6fba6a9240bd-openshift-service-ca\") pod \"c2507a90-02cd-4107-9df9-6fba6a9240bd\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " Nov 03 21:06:39.943376 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.941783 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-metrics-certs\") pod \"c2507a90-02cd-4107-9df9-6fba6a9240bd\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " Nov 03 21:06:39.943376 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.942041 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls\") pod \"c2507a90-02cd-4107-9df9-6fba6a9240bd\" (UID: \"c2507a90-02cd-4107-9df9-6fba6a9240bd\") " Nov 03 21:06:39.957529 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.956657 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2507a90-02cd-4107-9df9-6fba6a9240bd-openshift-service-ca" pod "c2507a90-02cd-4107-9df9-6fba6a9240bd" (UID: "c2507a90-02cd-4107-9df9-6fba6a9240bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:39.968338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.968301 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2507a90-02cd-4107-9df9-6fba6a9240bd-kube-api-access-j4f7l" pod "c2507a90-02cd-4107-9df9-6fba6a9240bd" (UID: "c2507a90-02cd-4107-9df9-6fba6a9240bd"). InnerVolumeSpecName "kube-api-access-j4f7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:39.968543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.968468 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-metrics-certs" pod "c2507a90-02cd-4107-9df9-6fba6a9240bd" (UID: "c2507a90-02cd-4107-9df9-6fba6a9240bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:39.991622 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:39.991559 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls" pod "c2507a90-02cd-4107-9df9-6fba6a9240bd" (UID: "c2507a90-02cd-4107-9df9-6fba6a9240bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:40.046183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.044662 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4f7l\" (UniqueName: \"kubernetes.io/projected/c2507a90-02cd-4107-9df9-6fba6a9240bd-kube-api-access-j4f7l\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:40.046183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.044698 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2507a90-02cd-4107-9df9-6fba6a9240bd-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:40.046183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.044714 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:40.046183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.044728 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2507a90-02cd-4107-9df9-6fba6a9240bd-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:40.306110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.306074 4227 generic.go:356] "Generic (PLEG): container finished" podID="c2507a90-02cd-4107-9df9-6fba6a9240bd" containerID="80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255" exitCode=0 Nov 03 21:06:40.306353 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.306122 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" Nov 03 21:06:40.306353 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.306174 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" event={"ID":"c2507a90-02cd-4107-9df9-6fba6a9240bd","Type":"ContainerDied","Data":"80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255"} Nov 03 21:06:40.306353 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.306241 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch" event={"ID":"c2507a90-02cd-4107-9df9-6fba6a9240bd","Type":"ContainerDied","Data":"f0dae2aca6da0c8a48e504388c6f4494a0dedb3f022d0478cd8d44f2b493070f"} Nov 03 21:06:40.306353 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.306270 4227 scope.go:122] "RemoveContainer" containerID="80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255" Nov 03 21:06:40.316277 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.316261 4227 scope.go:122] "RemoveContainer" containerID="80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255" Nov 03 21:06:40.316598 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:06:40.316573 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255\": container with ID starting with 80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255 not found: ID does not exist" containerID="80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255" Nov 03 21:06:40.316653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.316604 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255"} err="failed to get container status \"80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255\": rpc error: code = NotFound desc = could not find container \"80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255\": container with ID starting with 80fdfc05992f468e1b1b836bdc583b62e7c72c3293f1f67b70455af7852be255 not found: ID does not exist" Nov 03 21:06:40.509218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.499215 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch"] Nov 03 21:06:40.511413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.511377 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c63dc3-647c965b47-kw8ch"] Nov 03 21:06:40.900541 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:06:40.900509 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2507a90-02cd-4107-9df9-6fba6a9240bd" path="/var/lib/kubelet/pods/c2507a90-02cd-4107-9df9-6fba6a9240bd/volumes" Nov 03 21:06:56.086838 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:06:56.086795 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:07:26.082982 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:07:26.082944 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:07:52.825442 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:52.825354 4227 generic.go:356] "Generic (PLEG): container finished" podID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerID="83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975" exitCode=0 Nov 03 21:07:52.825442 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:52.825420 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" event={"ID":"9c84ab14-c774-41c4-852a-4eb8698c83ea","Type":"ContainerDied","Data":"83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975"} Nov 03 21:07:52.826016 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:52.825849 4227 scope.go:122] "RemoveContainer" containerID="83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975" Nov 03 21:07:52.827365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:52.827339 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:07:52.827473 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:52.827405 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:07:52.827473 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:52.827439 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:07:53.832590 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:53.832552 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" event={"ID":"9c84ab14-c774-41c4-852a-4eb8698c83ea","Type":"ContainerStarted","Data":"0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b"} Nov 03 21:07:54.703899 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:54.703870 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:07:54.704067 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:54.703999 4227 patch_prober.go:29] interesting pod/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:07:54.704114 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:54.704058 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 21:07:55.704357 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:55.704321 4227 patch_prober.go:29] interesting pod/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:07:55.704793 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:55.704390 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 21:07:56.106075 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:07:56.105976 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:07:56.704732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:56.704697 4227 patch_prober.go:29] interesting pod/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:07:56.705176 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:56.704750 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 21:07:57.704537 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:57.704501 4227 patch_prober.go:29] interesting pod/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:07:57.704820 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:57.704551 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 21:07:58.704496 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:58.704461 4227 patch_prober.go:29] interesting pod/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:07:58.704679 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:58.704515 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 21:07:59.709111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:59.709079 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:07:59.709634 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:59.709489 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:07:59.710216 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:07:59.710194 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:08:02.602878 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.602841 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz"] Nov 03 21:08:02.603497 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603473 4227 cpu_manager.go:414] "containerMap: removing container" podUID="6ec6f49d-4114-4945-929c-f5a6b4c70679" containerName="payload-processing" Nov 03 21:08:02.603497 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603495 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec6f49d-4114-4945-929c-f5a6b4c70679" containerName="payload-processing" Nov 03 21:08:02.603710 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603535 4227 cpu_manager.go:414] "containerMap: removing container" podUID="d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" containerName="payload-pre-processing" Nov 03 21:08:02.603710 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603543 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" containerName="payload-pre-processing" Nov 03 21:08:02.603710 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603573 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c2507a90-02cd-4107-9df9-6fba6a9240bd" containerName="maas-api" Nov 03 21:08:02.603710 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603580 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2507a90-02cd-4107-9df9-6fba6a9240bd" containerName="maas-api" Nov 03 21:08:02.603710 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603701 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2507a90-02cd-4107-9df9-6fba6a9240bd" containerName="maas-api" Nov 03 21:08:02.603953 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603723 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec6f49d-4114-4945-929c-f5a6b4c70679" containerName="payload-processing" Nov 03 21:08:02.603953 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.603738 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f6d2d1-a249-418c-9b6b-832ad3f7f67d" containerName="payload-pre-processing" Nov 03 21:08:02.606746 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.606726 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.702279 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.702238 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-211d68bf-openshift-default-dockercfg-c9mrc\"" Nov 03 21:08:02.705673 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.705641 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz"] Nov 03 21:08:02.747654 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.747618 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.747866 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.747661 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.747866 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.747683 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.747866 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.747813 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b1bba5-799f-451b-859c-741433af9d3c-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.747866 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.747847 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnrv\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-kube-api-access-zcnrv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.748088 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.747904 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.748088 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.748048 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.748293 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.748266 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.748334 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.748310 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b1bba5-799f-451b-859c-741433af9d3c-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.849960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.849922 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850141 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.849983 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b1bba5-799f-451b-859c-741433af9d3c-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850141 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850011 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnrv\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-kube-api-access-zcnrv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850141 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850060 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850336 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850230 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850338 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850370 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b1bba5-799f-451b-859c-741433af9d3c-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850438 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850465 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850492 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850784 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850761 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.850871 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.850848 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b1bba5-799f-451b-859c-741433af9d3c-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.851059 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.851032 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.851197 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.851067 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.852933 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.852876 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.853053 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.853032 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b1bba5-799f-451b-859c-741433af9d3c-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.903890 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.903862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnrv\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-kube-api-access-zcnrv\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.903890 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.903889 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-m4nlz\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:02.919063 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:02.919034 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:03.351225 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:03.351191 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz"] Nov 03 21:08:03.355828 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:08:03.355802 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b1bba5_799f_451b_859c_741433af9d3c.slice/crio-955337c1611bcc058e80ed81b63d47ae4f26cdfaade356bd59c3b4c000876734 WatchSource:0}: Error finding container 955337c1611bcc058e80ed81b63d47ae4f26cdfaade356bd59c3b4c000876734: Status 404 returned error can't find the container with id 955337c1611bcc058e80ed81b63d47ae4f26cdfaade356bd59c3b4c000876734 Nov 03 21:08:03.358680 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:03.358639 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:03.358771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:03.358730 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:03.358811 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:03.358775 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:03.910845 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:03.910805 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" event={"ID":"f9b1bba5-799f-451b-859c-741433af9d3c","Type":"ContainerStarted","Data":"e1ff6da966cd6322b739b717750e60a5573e63024e28faac63a752337380a21c"} Nov 03 21:08:03.911268 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:03.910865 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" event={"ID":"f9b1bba5-799f-451b-859c-741433af9d3c","Type":"ContainerStarted","Data":"955337c1611bcc058e80ed81b63d47ae4f26cdfaade356bd59c3b4c000876734"} Nov 03 21:08:03.920095 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:03.920068 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:04.073380 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:04.073319 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podStartSLOduration=2.073300526 podStartE2EDuration="2.073300526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:04.022754429 +0000 UTC m=+2351.259732306" watchObservedRunningTime="2026-11-03 21:08:04.073300526 +0000 UTC m=+2351.310278401" Nov 03 21:08:04.920976 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:04.920945 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:08:04.921388 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:04.920995 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:08:05.919929 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:05.919897 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:08:05.920103 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:05.919949 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:08:06.920273 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:06.920235 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:08:06.920778 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:06.920288 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:08:07.927127 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:07.927091 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:08:07.927697 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:07.927669 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:08:08.919961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:08.919926 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:08:08.920138 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:08.919975 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:08:09.920545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:09.920506 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:08:09.921198 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:09.921146 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:08:10.933840 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:10.933805 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:10.935371 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:10.935350 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:10.937345 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:10.937324 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:08:12.969810 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:12.969773 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr"] Nov 03 21:08:12.974874 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:12.974849 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.057430 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.057384 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-562eb2-openshift-default-dockercfg-fqtzk\"" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277381 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-envoy\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277465 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-token\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277545 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-certs\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277584 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-credential-socket\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277635 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ca7467-f49f-4151-85fc-5fc490e9a518-istiod-ca-cert\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277720 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-socket\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277772 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-podinfo\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277892 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-data\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.279491 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.277918 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvq7\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-kube-api-access-vfvq7\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379089 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-envoy\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379195 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-token\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379265 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-certs\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379299 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-credential-socket\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379341 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ca7467-f49f-4151-85fc-5fc490e9a518-istiod-ca-cert\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379413 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-socket\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379458 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-podinfo\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379597 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-data\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.379622 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvq7\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-kube-api-access-vfvq7\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.388183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.381209 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr"] Nov 03 21:08:13.398143 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.397112 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-envoy\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.398143 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.397654 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-socket\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.399722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.399685 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-certs\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.400428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.400266 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-data\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.401371 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.401315 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-credential-socket\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.464870 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.430862 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ca7467-f49f-4151-85fc-5fc490e9a518-istiod-ca-cert\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.465238 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.437840 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-podinfo\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.465399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.456985 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-token\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.479210 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.479130 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvq7\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-kube-api-access-vfvq7\") pod \"e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:13.625273 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:13.625176 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:14.065073 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:14.065035 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr"] Nov 03 21:08:14.069293 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:08:14.069263 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca7467_f49f_4151_85fc_5fc490e9a518.slice/crio-54a1124f04a25f2d137ac3a9db3e6a2d2be0043144e267ad48d756c5891d88a2 WatchSource:0}: Error finding container 54a1124f04a25f2d137ac3a9db3e6a2d2be0043144e267ad48d756c5891d88a2: Status 404 returned error can't find the container with id 54a1124f04a25f2d137ac3a9db3e6a2d2be0043144e267ad48d756c5891d88a2 Nov 03 21:08:14.072120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:14.072082 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:14.072240 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:14.072174 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:14.072240 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:14.072207 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:15.025039 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:15.025000 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" event={"ID":"f6ca7467-f49f-4151-85fc-5fc490e9a518","Type":"ContainerStarted","Data":"b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5"} Nov 03 21:08:15.025253 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:15.025060 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" event={"ID":"f6ca7467-f49f-4151-85fc-5fc490e9a518","Type":"ContainerStarted","Data":"54a1124f04a25f2d137ac3a9db3e6a2d2be0043144e267ad48d756c5891d88a2"} Nov 03 21:08:15.227864 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:15.227807 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" podStartSLOduration=3.227792485 podStartE2EDuration="3.227792485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:15.180080548 +0000 UTC m=+2362.417058421" watchObservedRunningTime="2026-11-03 21:08:15.227792485 +0000 UTC m=+2362.464770358" Nov 03 21:08:15.625778 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:15.625744 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:15.631892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:15.631867 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:16.031331 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:16.031297 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:16.032110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:16.032089 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:08:17.715399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.715364 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t"] Nov 03 21:08:17.718622 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.718601 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.763347 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.763318 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7"] Nov 03 21:08:17.767070 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.767049 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.814704 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.814674 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t"] Nov 03 21:08:17.845220 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845187 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c494a8ed-803e-4b5c-8854-960dac3d13fd-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.845506 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845419 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llcx7\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-kube-api-access-llcx7\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.845618 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845550 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjmt\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-kube-api-access-zfjmt\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.845676 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845618 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.845676 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845665 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.845774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845685 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.845774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845728 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.845774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845755 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.845974 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845951 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.846097 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.845989 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.846097 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846075 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.846256 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846114 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d08ec1-f534-45c2-933b-410a57767c15-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.846256 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846215 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.846366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846258 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d08ec1-f534-45c2-933b-410a57767c15-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.846366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846298 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.846366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846325 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.846517 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846377 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.846586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.846560 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.876959 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.876931 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7"] Nov 03 21:08:17.947843 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.947804 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948038 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.947885 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c494a8ed-803e-4b5c-8854-960dac3d13fd-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948038 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.947937 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcx7\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-kube-api-access-llcx7\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948038 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.947982 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjmt\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-kube-api-access-zfjmt\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948038 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948011 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948050 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948075 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948115 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948149 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948223 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948254 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948282 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948324 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d08ec1-f534-45c2-933b-410a57767c15-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948387 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948429 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d08ec1-f534-45c2-933b-410a57767c15-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.948682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948463 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948491 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948493 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.948682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948564 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.949046 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948744 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.949046 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.949022 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.949151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.948252 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.949349 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.949325 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.949413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.949335 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.949475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.949441 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c494a8ed-803e-4b5c-8854-960dac3d13fd-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.949725 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.949698 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.949800 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.949748 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.950412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.950387 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d08ec1-f534-45c2-933b-410a57767c15-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.953655 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.952426 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.953655 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.952685 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d08ec1-f534-45c2-933b-410a57767c15-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:17.953655 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.952776 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:17.954028 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:17.954000 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:18.011989 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.011913 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:18.012242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.012102 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcx7\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-kube-api-access-llcx7\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-7kf9t\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:18.012242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.012211 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjmt\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-kube-api-access-zfjmt\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:18.013417 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.013394 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-sdrv7\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:18.032863 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.032827 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:18.080212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.080180 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:18.495186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.484715 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t"] Nov 03 21:08:18.518189 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:08:18.503696 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc494a8ed_803e_4b5c_8854_960dac3d13fd.slice/crio-93667488a62e180b3ebb117e58e284d752bb00568b1cae695d6cebd64beb4ebb WatchSource:0}: Error finding container 93667488a62e180b3ebb117e58e284d752bb00568b1cae695d6cebd64beb4ebb: Status 404 returned error can't find the container with id 93667488a62e180b3ebb117e58e284d752bb00568b1cae695d6cebd64beb4ebb Nov 03 21:08:18.518189 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.508304 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:18.518189 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.508398 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:18.518189 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.508508 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:18.575510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.575471 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7"] Nov 03 21:08:18.608027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.606868 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:18.608027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.606969 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:18.608027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:18.607013 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:19.057634 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.057596 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" event={"ID":"c494a8ed-803e-4b5c-8854-960dac3d13fd","Type":"ContainerStarted","Data":"5666b8c36626bc912f50ffb98541e4d3bef331d6c5cdeb19faf75b719c73f9fb"} Nov 03 21:08:19.058134 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.057663 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" event={"ID":"c494a8ed-803e-4b5c-8854-960dac3d13fd","Type":"ContainerStarted","Data":"93667488a62e180b3ebb117e58e284d752bb00568b1cae695d6cebd64beb4ebb"} Nov 03 21:08:19.059425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.059400 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" event={"ID":"71d08ec1-f534-45c2-933b-410a57767c15","Type":"ContainerStarted","Data":"c447ff2fa5769aa0be63dcc70aa0f4ca8e403d0eff582b11ebe90c691ab96699"} Nov 03 21:08:19.059608 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.059446 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" event={"ID":"71d08ec1-f534-45c2-933b-410a57767c15","Type":"ContainerStarted","Data":"74ce720efbc9a38dcaae47cbeef77d98dd648702b04c7dee803b527602068d83"} Nov 03 21:08:19.081758 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.081735 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:19.083083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.083059 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:08:19.083227 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.083111 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:08:19.180182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.176910 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podStartSLOduration=2.176881109 podStartE2EDuration="2.176881109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:19.17258313 +0000 UTC m=+2366.409561033" watchObservedRunningTime="2026-11-03 21:08:19.176881109 +0000 UTC m=+2366.413858984" Nov 03 21:08:19.301123 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:19.300360 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podStartSLOduration=2.300335733 podStartE2EDuration="2.300335733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:19.281808952 +0000 UTC m=+2366.518786870" watchObservedRunningTime="2026-11-03 21:08:19.300335733 +0000 UTC m=+2366.537313674" Nov 03 21:08:20.033558 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:20.033528 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:20.034883 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:20.034849 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:08:20.035058 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:20.035027 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:08:20.103044 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:20.103001 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:08:20.103551 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:20.103063 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:08:21.033769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:21.033734 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:08:21.033957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:21.033786 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:08:21.086186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:21.085182 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:08:21.086186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:21.085255 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:08:22.144111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:22.143423 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:08:22.144111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:22.143478 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:08:22.177420 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:22.173939 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:08:22.177420 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:22.174011 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:08:22.575179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:22.574724 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx"] Nov 03 21:08:22.575179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:22.574983 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" podUID="9e50bbbc-e6d8-4381-b4e2-bee22edb8620" containerName="maas-api" containerID="cri-o://e7f5d8af99467fb5cda4c769d9b406a86e304ebf20a3a986696483ddb73959c5" gracePeriod=30 Nov 03 21:08:22.824183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:22.812303 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" podUID="9e50bbbc-e6d8-4381-b4e2-bee22edb8620" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.97:8443/health\": dial tcp 10.134.0.97:8443: connect: connection refused" Nov 03 21:08:23.065178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.056360 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:08:23.065178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.056449 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:08:23.091202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.089715 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:08:23.091202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.089775 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:08:23.277310 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.277274 4227 generic.go:356] "Generic (PLEG): container finished" podID="9e50bbbc-e6d8-4381-b4e2-bee22edb8620" containerID="e7f5d8af99467fb5cda4c769d9b406a86e304ebf20a3a986696483ddb73959c5" exitCode=0 Nov 03 21:08:23.289179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.277907 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" event={"ID":"9e50bbbc-e6d8-4381-b4e2-bee22edb8620","Type":"ContainerDied","Data":"e7f5d8af99467fb5cda4c769d9b406a86e304ebf20a3a986696483ddb73959c5"} Nov 03 21:08:23.931147 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.931118 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:08:23.932578 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.932553 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls\") pod \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " Nov 03 21:08:23.932725 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.932593 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-kube-api-access-2vtcq\" (UniqueName: \"kubernetes.io/projected/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-kube-api-access-2vtcq\") pod \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " Nov 03 21:08:23.932725 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.932622 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-openshift-service-ca\") pod \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " Nov 03 21:08:23.932860 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.932749 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-metrics-certs\") pod \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\" (UID: \"9e50bbbc-e6d8-4381-b4e2-bee22edb8620\") " Nov 03 21:08:23.933714 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.933684 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-openshift-service-ca" pod "9e50bbbc-e6d8-4381-b4e2-bee22edb8620" (UID: "9e50bbbc-e6d8-4381-b4e2-bee22edb8620"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:23.935720 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.935692 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-metrics-certs" pod "9e50bbbc-e6d8-4381-b4e2-bee22edb8620" (UID: "9e50bbbc-e6d8-4381-b4e2-bee22edb8620"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:23.936036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.936016 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-kube-api-access-2vtcq" pod "9e50bbbc-e6d8-4381-b4e2-bee22edb8620" (UID: "9e50bbbc-e6d8-4381-b4e2-bee22edb8620"). InnerVolumeSpecName "kube-api-access-2vtcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:23.937958 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:23.937934 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls" pod "9e50bbbc-e6d8-4381-b4e2-bee22edb8620" (UID: "9e50bbbc-e6d8-4381-b4e2-bee22edb8620"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:24.033264 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.033230 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:08:24.033529 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.033493 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:08:24.034006 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.033978 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:24.034104 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.034011 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtcq\" (UniqueName: \"kubernetes.io/projected/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-kube-api-access-2vtcq\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:24.034104 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.034026 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:24.034104 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.034041 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e50bbbc-e6d8-4381-b4e2-bee22edb8620-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:24.108280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.106044 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:08:24.108280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.106102 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:08:24.444399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.444365 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" Nov 03 21:08:24.444831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.444369 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx" event={"ID":"9e50bbbc-e6d8-4381-b4e2-bee22edb8620","Type":"ContainerDied","Data":"3489655aef9368506d9d9a65679f6e2fd96f23cd0aa2ddc71d5371d058717814"} Nov 03 21:08:24.444831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.444509 4227 scope.go:122] "RemoveContainer" containerID="e7f5d8af99467fb5cda4c769d9b406a86e304ebf20a3a986696483ddb73959c5" Nov 03 21:08:24.625700 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.625667 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx"] Nov 03 21:08:24.631183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.631143 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-46ca0b-8498ccd8b5-vhdnx"] Nov 03 21:08:24.921608 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:24.921574 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e50bbbc-e6d8-4381-b4e2-bee22edb8620" path="/var/lib/kubelet/pods/9e50bbbc-e6d8-4381-b4e2-bee22edb8620/volumes" Nov 03 21:08:25.033738 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:25.033700 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:08:25.033921 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:25.033783 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:08:25.080631 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:25.080593 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:08:25.080897 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:25.080872 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 21:08:26.033346 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:26.033306 4227 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:08:26.033866 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:26.033387 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 21:08:26.107473 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:08:26.106819 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:08:26.350317 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:26.350222 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:26.350856 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:26.350831 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:26.351923 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:26.351894 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:08:27.222180 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.221560 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t"] Nov 03 21:08:27.223236 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.223132 4227 cpu_manager.go:414] "containerMap: removing container" podUID="9e50bbbc-e6d8-4381-b4e2-bee22edb8620" containerName="maas-api" Nov 03 21:08:27.225713 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.223369 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e50bbbc-e6d8-4381-b4e2-bee22edb8620" containerName="maas-api" Nov 03 21:08:27.225713 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.223553 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e50bbbc-e6d8-4381-b4e2-bee22edb8620" containerName="maas-api" Nov 03 21:08:27.229503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.228696 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.239069 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.239026 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t"] Nov 03 21:08:27.273539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.272263 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:27.273539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.273392 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:27.278657 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.278587 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:08:27.282402 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.281689 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-98d1d2-openshift-default-dockercfg-nsfq2\"" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.408544 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6278553-4b48-4f8c-bbc3-d3217718265c-istiod-ca-cert\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.408628 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-credential-socket\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.408663 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6pn\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-kube-api-access-9x6pn\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.408808 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-data\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.408850 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-podinfo\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.408884 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-socket\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.409000 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-envoy\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.409042 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-token\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.414195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.409067 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-certs\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522103 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-data\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522220 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-podinfo\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522261 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-socket\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522391 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-envoy\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522437 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-token\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522466 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-certs\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522556 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6278553-4b48-4f8c-bbc3-d3217718265c-istiod-ca-cert\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522628 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-credential-socket\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.522661 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6pn\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-kube-api-access-9x6pn\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.523672 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-data\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.523920 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-certs\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.524597 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6278553-4b48-4f8c-bbc3-d3217718265c-istiod-ca-cert\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.524910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.524617 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-credential-socket\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.525770 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.525077 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-socket\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.556428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.556393 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-envoy\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.559200 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.559171 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-podinfo\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.579759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.579724 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-token\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.592895 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.592847 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6pn\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-kube-api-access-9x6pn\") pod \"e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:27.613569 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:27.613531 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:28.067231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.067195 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t"] Nov 03 21:08:28.073017 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:08:28.072986 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6278553_4b48_4f8c_bbc3_d3217718265c.slice/crio-2253aa93da44d16739e0b9cb18111bb4772d116b46b1f42bbbc5396badda6ed6 WatchSource:0}: Error finding container 2253aa93da44d16739e0b9cb18111bb4772d116b46b1f42bbbc5396badda6ed6: Status 404 returned error can't find the container with id 2253aa93da44d16739e0b9cb18111bb4772d116b46b1f42bbbc5396badda6ed6 Nov 03 21:08:28.076522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.076483 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:28.076601 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.076589 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:28.076660 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.076645 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:08:28.588778 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.588743 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" event={"ID":"f6278553-4b48-4f8c-bbc3-d3217718265c","Type":"ContainerStarted","Data":"7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f"} Nov 03 21:08:28.589278 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.588836 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" event={"ID":"f6278553-4b48-4f8c-bbc3-d3217718265c","Type":"ContainerStarted","Data":"2253aa93da44d16739e0b9cb18111bb4772d116b46b1f42bbbc5396badda6ed6"} Nov 03 21:08:28.614373 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.614343 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:28.750263 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:28.750202 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podStartSLOduration=1.750185561 podStartE2EDuration="1.750185561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:28.745571755 +0000 UTC m=+2375.982549626" watchObservedRunningTime="2026-11-03 21:08:28.750185561 +0000 UTC m=+2375.987163435" Nov 03 21:08:29.620142 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:29.620113 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:30.603356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:30.603315 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:30.604290 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:30.604268 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:08:53.179842 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:53.179814 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:08:53.185424 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:53.185396 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:08:56.087855 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:56.087818 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j"] Nov 03 21:08:56.088387 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:08:56.088301 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:08:56.200379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:56.200343 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s"] Nov 03 21:08:57.028979 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:08:57.028947 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5"] Nov 03 21:09:01.088084 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.088039 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" podUID="0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" containerName="payload-processing" containerID="cri-o://21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8" gracePeriod=25 Nov 03 21:09:01.201398 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.201351 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" podUID="2c642461-3a57-4cca-89d1-20caf9f69d44" containerName="payload-pre-processing" containerID="cri-o://34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb" gracePeriod=25 Nov 03 21:09:01.377391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.377364 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:09:01.471273 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.471238 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-kube-api-access-bh8cg\" (UniqueName: \"kubernetes.io/projected/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-kube-api-access-bh8cg\") pod \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " Nov 03 21:09:01.471620 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.471603 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-plugins-config-volume\") pod \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\" (UID: \"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c\") " Nov 03 21:09:01.472046 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.472018 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-plugins-config-volume" pod "0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" (UID: "0ec8860f-fba0-4a16-b9c6-c155a51f9a2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:01.473592 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.473565 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-kube-api-access-bh8cg" pod "0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" (UID: "0ec8860f-fba0-4a16-b9c6-c155a51f9a2c"). InnerVolumeSpecName "kube-api-access-bh8cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:01.499265 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.499237 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:09:01.574125 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.574088 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:01.574125 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.574127 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh8cg\" (UniqueName: \"kubernetes.io/projected/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c-kube-api-access-bh8cg\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:01.675201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.675153 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c642461-3a57-4cca-89d1-20caf9f69d44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c642461-3a57-4cca-89d1-20caf9f69d44-plugins-config-volume\") pod \"2c642461-3a57-4cca-89d1-20caf9f69d44\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " Nov 03 21:09:01.675402 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.675268 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c642461-3a57-4cca-89d1-20caf9f69d44-kube-api-access-4w8f9\" (UniqueName: \"kubernetes.io/projected/2c642461-3a57-4cca-89d1-20caf9f69d44-kube-api-access-4w8f9\") pod \"2c642461-3a57-4cca-89d1-20caf9f69d44\" (UID: \"2c642461-3a57-4cca-89d1-20caf9f69d44\") " Nov 03 21:09:01.675598 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.675573 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c642461-3a57-4cca-89d1-20caf9f69d44-plugins-config-volume" pod "2c642461-3a57-4cca-89d1-20caf9f69d44" (UID: "2c642461-3a57-4cca-89d1-20caf9f69d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:01.675832 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.675814 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c642461-3a57-4cca-89d1-20caf9f69d44-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:01.677389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.677368 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c642461-3a57-4cca-89d1-20caf9f69d44-kube-api-access-4w8f9" pod "2c642461-3a57-4cca-89d1-20caf9f69d44" (UID: "2c642461-3a57-4cca-89d1-20caf9f69d44"). InnerVolumeSpecName "kube-api-access-4w8f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:01.777084 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:01.777048 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8f9\" (UniqueName: \"kubernetes.io/projected/2c642461-3a57-4cca-89d1-20caf9f69d44-kube-api-access-4w8f9\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:02.196998 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.196954 4227 generic.go:356] "Generic (PLEG): container finished" podID="2c642461-3a57-4cca-89d1-20caf9f69d44" containerID="34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb" exitCode=0 Nov 03 21:09:02.197432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.197099 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" Nov 03 21:09:02.197528 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.197121 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" event={"ID":"2c642461-3a57-4cca-89d1-20caf9f69d44","Type":"ContainerDied","Data":"34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb"} Nov 03 21:09:02.197618 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.197560 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s" event={"ID":"2c642461-3a57-4cca-89d1-20caf9f69d44","Type":"ContainerDied","Data":"41e81562614c3938dcf928db8f9e969174578b771fceb306a3b1c24fd3ceddd6"} Nov 03 21:09:02.197682 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.197667 4227 scope.go:122] "RemoveContainer" containerID="34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb" Nov 03 21:09:02.213719 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.212429 4227 generic.go:356] "Generic (PLEG): container finished" podID="0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" containerID="21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8" exitCode=0 Nov 03 21:09:02.213719 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.212557 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" event={"ID":"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c","Type":"ContainerDied","Data":"21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8"} Nov 03 21:09:02.213719 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.212594 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" event={"ID":"0ec8860f-fba0-4a16-b9c6-c155a51f9a2c","Type":"ContainerDied","Data":"adc1993bf3ac2578f81699a946ad38d26ee0f65687406c019604f916d51634dd"} Nov 03 21:09:02.213719 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.212663 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j" Nov 03 21:09:02.216878 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.216853 4227 scope.go:122] "RemoveContainer" containerID="34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb" Nov 03 21:09:02.217151 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:02.217123 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb\": container with ID starting with 34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb not found: ID does not exist" containerID="34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb" Nov 03 21:09:02.217289 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.217175 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb"} err="failed to get container status \"34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb\": rpc error: code = NotFound desc = could not find container \"34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb\": container with ID starting with 34854dc0fc73c51c4b31d20f92ca9946933755e2b549f6a1795ad231da18bedb not found: ID does not exist" Nov 03 21:09:02.217289 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.217200 4227 scope.go:122] "RemoveContainer" containerID="21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8" Nov 03 21:09:02.227645 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.227616 4227 scope.go:122] "RemoveContainer" containerID="21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8" Nov 03 21:09:02.227945 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:02.227925 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8\": container with ID starting with 21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8 not found: ID does not exist" containerID="21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8" Nov 03 21:09:02.227997 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.227956 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8"} err="failed to get container status \"21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8\": rpc error: code = NotFound desc = could not find container \"21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8\": container with ID starting with 21a411272958a24b35401465d734ede965703b8179b8447736623e147142cde8 not found: ID does not exist" Nov 03 21:09:02.354175 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.354124 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j"] Nov 03 21:09:02.360764 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.360690 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3ca125-754bf5b8c9-5rg8j"] Nov 03 21:09:02.460522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.460425 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s"] Nov 03 21:09:02.463598 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.463567 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3ca125-786b45c879-q296s"] Nov 03 21:09:02.901309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.901265 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" path="/var/lib/kubelet/pods/0ec8860f-fba0-4a16-b9c6-c155a51f9a2c/volumes" Nov 03 21:09:02.901837 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:02.901815 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c642461-3a57-4cca-89d1-20caf9f69d44" path="/var/lib/kubelet/pods/2c642461-3a57-4cca-89d1-20caf9f69d44/volumes" Nov 03 21:09:10.496092 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:10.496054 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8"] Nov 03 21:09:12.051711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.051666 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" containerID="cri-o://4760c0e10469cba29c4d11b7751d2ec64762e5e983be0292cbf6b66b08076767" gracePeriod=45 Nov 03 21:09:12.110313 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.110274 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused" Nov 03 21:09:12.279497 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.279461 4227 generic.go:356] "Generic (PLEG): container finished" podID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerID="4760c0e10469cba29c4d11b7751d2ec64762e5e983be0292cbf6b66b08076767" exitCode=0 Nov 03 21:09:12.279671 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.279563 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" event={"ID":"e0ad67a2-6719-4c18-9f54-fc3633bccc78","Type":"ContainerDied","Data":"4760c0e10469cba29c4d11b7751d2ec64762e5e983be0292cbf6b66b08076767"} Nov 03 21:09:12.327653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.327630 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 21:09:12.483584 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.483545 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ad67a2-6719-4c18-9f54-fc3633bccc78-kube-api-access-kjzd2\" (UniqueName: \"kubernetes.io/projected/e0ad67a2-6719-4c18-9f54-fc3633bccc78-kube-api-access-kjzd2\") pod \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " Nov 03 21:09:12.483780 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.483690 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-dshm\") pod \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " Nov 03 21:09:12.483850 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.483778 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tls-certs\") pod \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " Nov 03 21:09:12.483850 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.483822 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tmp-dir\") pod \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " Nov 03 21:09:12.483962 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.483874 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-model-cache\") pod \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " Nov 03 21:09:12.483962 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.483904 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-home\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-home\") pod \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\" (UID: \"e0ad67a2-6719-4c18-9f54-fc3633bccc78\") " Nov 03 21:09:12.484710 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.484685 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-home" pod "e0ad67a2-6719-4c18-9f54-fc3633bccc78" (UID: "e0ad67a2-6719-4c18-9f54-fc3633bccc78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.486704 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.486683 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tmp-dir" pod "e0ad67a2-6719-4c18-9f54-fc3633bccc78" (UID: "e0ad67a2-6719-4c18-9f54-fc3633bccc78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.487004 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.486987 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-model-cache" pod "e0ad67a2-6719-4c18-9f54-fc3633bccc78" (UID: "e0ad67a2-6719-4c18-9f54-fc3633bccc78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.488406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.488367 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ad67a2-6719-4c18-9f54-fc3633bccc78-kube-api-access-kjzd2" pod "e0ad67a2-6719-4c18-9f54-fc3633bccc78" (UID: "e0ad67a2-6719-4c18-9f54-fc3633bccc78"). InnerVolumeSpecName "kube-api-access-kjzd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:12.488846 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.488812 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-dshm" pod "e0ad67a2-6719-4c18-9f54-fc3633bccc78" (UID: "e0ad67a2-6719-4c18-9f54-fc3633bccc78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:12.489451 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.489420 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tls-certs" pod "e0ad67a2-6719-4c18-9f54-fc3633bccc78" (UID: "e0ad67a2-6719-4c18-9f54-fc3633bccc78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:12.585530 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.585445 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.585530 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.585486 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.585530 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.585501 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.585530 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.585516 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjzd2\" (UniqueName: \"kubernetes.io/projected/e0ad67a2-6719-4c18-9f54-fc3633bccc78-kube-api-access-kjzd2\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.585530 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.585530 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ad67a2-6719-4c18-9f54-fc3633bccc78-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:12.585804 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:12.585543 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ad67a2-6719-4c18-9f54-fc3633bccc78-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:13.287759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:13.287678 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" Nov 03 21:09:13.288224 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:13.287675 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5" event={"ID":"e0ad67a2-6719-4c18-9f54-fc3633bccc78","Type":"ContainerDied","Data":"aa1010953088d4216ae7670e40d7b2d3b5c2a6ebd0f069019044ac9b706bf2ca"} Nov 03 21:09:13.288224 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:13.287809 4227 scope.go:122] "RemoveContainer" containerID="4760c0e10469cba29c4d11b7751d2ec64762e5e983be0292cbf6b66b08076767" Nov 03 21:09:13.441930 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:13.441893 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5"] Nov 03 21:09:13.489810 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:13.489778 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a19980/distinct-w6-a19980-kserve-6649db57f9-v2tm5"] Nov 03 21:09:14.902103 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:14.902066 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" path="/var/lib/kubelet/pods/e0ad67a2-6719-4c18-9f54-fc3633bccc78/volumes" Nov 03 21:09:25.516760 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.516716 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" containerID="cri-o://47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306" gracePeriod=45 Nov 03 21:09:25.781304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.781233 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 21:09:25.871012 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.870984 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-tmp-dir\") pod \"aff579ec-c609-4092-b348-6f4c24e2dc93\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " Nov 03 21:09:25.871231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871088 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-dshm\") pod \"aff579ec-c609-4092-b348-6f4c24e2dc93\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " Nov 03 21:09:25.871231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871119 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-model-cache\") pod \"aff579ec-c609-4092-b348-6f4c24e2dc93\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " Nov 03 21:09:25.871231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871198 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff579ec-c609-4092-b348-6f4c24e2dc93-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff579ec-c609-4092-b348-6f4c24e2dc93-tls-certs\") pod \"aff579ec-c609-4092-b348-6f4c24e2dc93\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " Nov 03 21:09:25.871428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871276 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-home\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-home\") pod \"aff579ec-c609-4092-b348-6f4c24e2dc93\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " Nov 03 21:09:25.871428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871300 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-tmp-dir" pod "aff579ec-c609-4092-b348-6f4c24e2dc93" (UID: "aff579ec-c609-4092-b348-6f4c24e2dc93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.871428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871348 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff579ec-c609-4092-b348-6f4c24e2dc93-kube-api-access-djzt6\" (UniqueName: \"kubernetes.io/projected/aff579ec-c609-4092-b348-6f4c24e2dc93-kube-api-access-djzt6\") pod \"aff579ec-c609-4092-b348-6f4c24e2dc93\" (UID: \"aff579ec-c609-4092-b348-6f4c24e2dc93\") " Nov 03 21:09:25.871589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871430 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-model-cache" pod "aff579ec-c609-4092-b348-6f4c24e2dc93" (UID: "aff579ec-c609-4092-b348-6f4c24e2dc93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.871589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871549 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-home" pod "aff579ec-c609-4092-b348-6f4c24e2dc93" (UID: "aff579ec-c609-4092-b348-6f4c24e2dc93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.871825 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871807 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.871891 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871829 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.871891 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.871843 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.873529 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.873502 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff579ec-c609-4092-b348-6f4c24e2dc93-tls-certs" pod "aff579ec-c609-4092-b348-6f4c24e2dc93" (UID: "aff579ec-c609-4092-b348-6f4c24e2dc93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:25.873658 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.873548 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff579ec-c609-4092-b348-6f4c24e2dc93-kube-api-access-djzt6" pod "aff579ec-c609-4092-b348-6f4c24e2dc93" (UID: "aff579ec-c609-4092-b348-6f4c24e2dc93"). InnerVolumeSpecName "kube-api-access-djzt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:25.873658 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.873621 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-dshm" pod "aff579ec-c609-4092-b348-6f4c24e2dc93" (UID: "aff579ec-c609-4092-b348-6f4c24e2dc93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:25.973791 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.973761 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff579ec-c609-4092-b348-6f4c24e2dc93-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.973791 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.973791 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzt6\" (UniqueName: \"kubernetes.io/projected/aff579ec-c609-4092-b348-6f4c24e2dc93-kube-api-access-djzt6\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:25.973984 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:25.973807 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff579ec-c609-4092-b348-6f4c24e2dc93-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:26.100306 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:26.100201 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:09:26.400732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.400694 4227 generic.go:356] "Generic (PLEG): container finished" podID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerID="47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306" exitCode=0 Nov 03 21:09:26.401082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.401057 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" event={"ID":"aff579ec-c609-4092-b348-6f4c24e2dc93","Type":"ContainerDied","Data":"47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306"} Nov 03 21:09:26.401268 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.401251 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" event={"ID":"aff579ec-c609-4092-b348-6f4c24e2dc93","Type":"ContainerDied","Data":"0db6d2d79e7b8aca932ccf52b4a23910db73d4cec7b9d99e709855f08add42f0"} Nov 03 21:09:26.401389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.401377 4227 scope.go:122] "RemoveContainer" containerID="47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306" Nov 03 21:09:26.401665 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.401645 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8" Nov 03 21:09:26.413541 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.413518 4227 scope.go:122] "RemoveContainer" containerID="47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306" Nov 03 21:09:26.413817 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:26.413794 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306\": container with ID starting with 47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306 not found: ID does not exist" containerID="47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306" Nov 03 21:09:26.413870 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.413823 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306"} err="failed to get container status \"47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306\": rpc error: code = NotFound desc = could not find container \"47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306\": container with ID starting with 47de979270021f6a851eb89c69de9658c4bc44450192f4ed43ee357f12b22306 not found: ID does not exist" Nov 03 21:09:26.557902 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.557821 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8"] Nov 03 21:09:26.603424 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.603390 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/premium-w5-1ec892-kserve-7df765b78c-bltv8"] Nov 03 21:09:26.901145 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:26.901110 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" path="/var/lib/kubelet/pods/aff579ec-c609-4092-b348-6f4c24e2dc93/volumes" Nov 03 21:09:29.228294 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:29.228260 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5"] Nov 03 21:09:37.517404 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:37.517366 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g"] Nov 03 21:09:37.517910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:37.517607 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" podUID="b8ce8e3d-bf70-46a2-b705-f52e42403c79" containerName="maas-api" containerID="cri-o://efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec" gracePeriod=30 Nov 03 21:09:37.992177 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:37.991836 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:09:38.085304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.085270 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ce8e3d-bf70-46a2-b705-f52e42403c79-openshift-service-ca" pod "b8ce8e3d-bf70-46a2-b705-f52e42403c79" (UID: "b8ce8e3d-bf70-46a2-b705-f52e42403c79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:38.085548 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.085529 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ce8e3d-bf70-46a2-b705-f52e42403c79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce8e3d-bf70-46a2-b705-f52e42403c79-openshift-service-ca\") pod \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " Nov 03 21:09:38.085765 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.085741 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls\") pod \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " Nov 03 21:09:38.086439 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.086419 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-metrics-certs\") pod \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " Nov 03 21:09:38.086960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.086939 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ce8e3d-bf70-46a2-b705-f52e42403c79-kube-api-access-6vvmq\" (UniqueName: \"kubernetes.io/projected/b8ce8e3d-bf70-46a2-b705-f52e42403c79-kube-api-access-6vvmq\") pod \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\" (UID: \"b8ce8e3d-bf70-46a2-b705-f52e42403c79\") " Nov 03 21:09:38.087898 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.087876 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce8e3d-bf70-46a2-b705-f52e42403c79-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.091376 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.091336 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-metrics-certs" pod "b8ce8e3d-bf70-46a2-b705-f52e42403c79" (UID: "b8ce8e3d-bf70-46a2-b705-f52e42403c79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:38.091650 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.091623 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls" pod "b8ce8e3d-bf70-46a2-b705-f52e42403c79" (UID: "b8ce8e3d-bf70-46a2-b705-f52e42403c79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:38.093740 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.093717 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ce8e3d-bf70-46a2-b705-f52e42403c79-kube-api-access-6vvmq" pod "b8ce8e3d-bf70-46a2-b705-f52e42403c79" (UID: "b8ce8e3d-bf70-46a2-b705-f52e42403c79"). InnerVolumeSpecName "kube-api-access-6vvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:38.189478 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.189439 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.189478 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.189469 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vvmq\" (UniqueName: \"kubernetes.io/projected/b8ce8e3d-bf70-46a2-b705-f52e42403c79-kube-api-access-6vvmq\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.189478 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.189479 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce8e3d-bf70-46a2-b705-f52e42403c79-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.516232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.513282 4227 generic.go:356] "Generic (PLEG): container finished" podID="b8ce8e3d-bf70-46a2-b705-f52e42403c79" containerID="efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec" exitCode=0 Nov 03 21:09:38.516232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.513398 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" event={"ID":"b8ce8e3d-bf70-46a2-b705-f52e42403c79","Type":"ContainerDied","Data":"efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec"} Nov 03 21:09:38.516232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.513452 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" event={"ID":"b8ce8e3d-bf70-46a2-b705-f52e42403c79","Type":"ContainerDied","Data":"2628451ca8ea20aaed726812abfc3b60e3d4f4a86b440e510dc3d1a5e5fe17d0"} Nov 03 21:09:38.516232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.513481 4227 scope.go:122] "RemoveContainer" containerID="efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec" Nov 03 21:09:38.516232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.513634 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g" Nov 03 21:09:38.607544 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.607381 4227 scope.go:122] "RemoveContainer" containerID="efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec" Nov 03 21:09:38.607981 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:38.607853 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec\": container with ID starting with efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec not found: ID does not exist" containerID="efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec" Nov 03 21:09:38.607981 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.607910 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec"} err="failed to get container status \"efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec\": rpc error: code = NotFound desc = could not find container \"efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec\": container with ID starting with efb0654871b61167f22b35415af97b167b48958657c1504ca86dd5ac8be300ec not found: ID does not exist" Nov 03 21:09:38.705456 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.705384 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g"] Nov 03 21:09:38.751784 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.751749 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c571ef-69b96c47d9-pgl7g"] Nov 03 21:09:38.939332 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:38.939293 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ce8e3d-bf70-46a2-b705-f52e42403c79" path="/var/lib/kubelet/pods/b8ce8e3d-bf70-46a2-b705-f52e42403c79/volumes" Nov 03 21:09:44.250691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.250630 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" podUID="7e32d66d-1706-4c7f-8465-727dc51387f9" containerName="main" containerID="cri-o://4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168" gracePeriod=45 Nov 03 21:09:44.522343 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.522312 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:09:44.629220 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629183 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-tmp-dir\") pod \"7e32d66d-1706-4c7f-8465-727dc51387f9\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " Nov 03 21:09:44.629428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629285 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e32d66d-1706-4c7f-8465-727dc51387f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e32d66d-1706-4c7f-8465-727dc51387f9-tls-certs\") pod \"7e32d66d-1706-4c7f-8465-727dc51387f9\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " Nov 03 21:09:44.629428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629314 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-dshm\") pod \"7e32d66d-1706-4c7f-8465-727dc51387f9\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " Nov 03 21:09:44.629428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629353 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-home\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-home\") pod \"7e32d66d-1706-4c7f-8465-727dc51387f9\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " Nov 03 21:09:44.629428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629404 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-tmp-dir" pod "7e32d66d-1706-4c7f-8465-727dc51387f9" (UID: "7e32d66d-1706-4c7f-8465-727dc51387f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:44.629428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629421 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-model-cache\") pod \"7e32d66d-1706-4c7f-8465-727dc51387f9\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " Nov 03 21:09:44.629726 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629471 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e32d66d-1706-4c7f-8465-727dc51387f9-kube-api-access-87psz\" (UniqueName: \"kubernetes.io/projected/7e32d66d-1706-4c7f-8465-727dc51387f9-kube-api-access-87psz\") pod \"7e32d66d-1706-4c7f-8465-727dc51387f9\" (UID: \"7e32d66d-1706-4c7f-8465-727dc51387f9\") " Nov 03 21:09:44.629726 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629683 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-home" pod "7e32d66d-1706-4c7f-8465-727dc51387f9" (UID: "7e32d66d-1706-4c7f-8465-727dc51387f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:44.629926 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.629902 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-model-cache" pod "7e32d66d-1706-4c7f-8465-727dc51387f9" (UID: "7e32d66d-1706-4c7f-8465-727dc51387f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:44.630228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.630205 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:44.630338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.630229 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:44.630338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.630244 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:44.631861 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.631839 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-dshm" pod "7e32d66d-1706-4c7f-8465-727dc51387f9" (UID: "7e32d66d-1706-4c7f-8465-727dc51387f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:44.632031 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.632014 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e32d66d-1706-4c7f-8465-727dc51387f9-kube-api-access-87psz" pod "7e32d66d-1706-4c7f-8465-727dc51387f9" (UID: "7e32d66d-1706-4c7f-8465-727dc51387f9"). InnerVolumeSpecName "kube-api-access-87psz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:44.632481 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.632464 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e32d66d-1706-4c7f-8465-727dc51387f9-tls-certs" pod "7e32d66d-1706-4c7f-8465-727dc51387f9" (UID: "7e32d66d-1706-4c7f-8465-727dc51387f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:44.643397 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.643244 4227 generic.go:356] "Generic (PLEG): container finished" podID="7e32d66d-1706-4c7f-8465-727dc51387f9" containerID="4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168" exitCode=0 Nov 03 21:09:44.643509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.643441 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" event={"ID":"7e32d66d-1706-4c7f-8465-727dc51387f9","Type":"ContainerDied","Data":"4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168"} Nov 03 21:09:44.643509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.643475 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" event={"ID":"7e32d66d-1706-4c7f-8465-727dc51387f9","Type":"ContainerDied","Data":"8c3249a11f458d835cd7f8b74d557fa4cd17382152dcb6dcc42a60c5032fa7aa"} Nov 03 21:09:44.643509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.643492 4227 scope.go:122] "RemoveContainer" containerID="4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168" Nov 03 21:09:44.643683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.643617 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5" Nov 03 21:09:44.672336 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.670486 4227 scope.go:122] "RemoveContainer" containerID="4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168" Nov 03 21:09:44.672336 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:44.670815 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168\": container with ID starting with 4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168 not found: ID does not exist" containerID="4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168" Nov 03 21:09:44.672336 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.670851 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168"} err="failed to get container status \"4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168\": rpc error: code = NotFound desc = could not find container \"4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168\": container with ID starting with 4f91d4b8d0dc604484b9e62ca926d669e309d943b541919b517cee60e30cb168 not found: ID does not exist" Nov 03 21:09:44.731014 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.730981 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e32d66d-1706-4c7f-8465-727dc51387f9-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:44.731014 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.731009 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e32d66d-1706-4c7f-8465-727dc51387f9-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:44.731014 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.731019 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87psz\" (UniqueName: \"kubernetes.io/projected/7e32d66d-1706-4c7f-8465-727dc51387f9-kube-api-access-87psz\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:44.804571 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.804488 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5"] Nov 03 21:09:44.808713 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.808683 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-211d68bf/e2e-lifecycle-model-211d68bf-kserve-7bdc7cd7c7-b6dh5"] Nov 03 21:09:44.901930 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:44.901896 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e32d66d-1706-4c7f-8465-727dc51387f9" path="/var/lib/kubelet/pods/7e32d66d-1706-4c7f-8465-727dc51387f9/volumes" Nov 03 21:09:49.037811 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.037769 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw"] Nov 03 21:09:49.038298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.038019 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" podUID="1f0c46b1-d75f-4f7d-b10d-5899d1770e64" containerName="maas-api" containerID="cri-o://de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f" gracePeriod=30 Nov 03 21:09:49.472696 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.472373 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:09:49.542229 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.541699 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls\") pod \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " Nov 03 21:09:49.542627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.542607 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-metrics-certs\") pod \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " Nov 03 21:09:49.543626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.543225 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-openshift-service-ca\") pod \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " Nov 03 21:09:49.543916 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.543886 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-kube-api-access-jrzgn\" (UniqueName: \"kubernetes.io/projected/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-kube-api-access-jrzgn\") pod \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\" (UID: \"1f0c46b1-d75f-4f7d-b10d-5899d1770e64\") " Nov 03 21:09:49.547624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.543594 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-openshift-service-ca" pod "1f0c46b1-d75f-4f7d-b10d-5899d1770e64" (UID: "1f0c46b1-d75f-4f7d-b10d-5899d1770e64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:49.561681 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.561624 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-kube-api-access-jrzgn" pod "1f0c46b1-d75f-4f7d-b10d-5899d1770e64" (UID: "1f0c46b1-d75f-4f7d-b10d-5899d1770e64"). InnerVolumeSpecName "kube-api-access-jrzgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:49.561681 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.561658 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls" pod "1f0c46b1-d75f-4f7d-b10d-5899d1770e64" (UID: "1f0c46b1-d75f-4f7d-b10d-5899d1770e64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:49.572369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.564450 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-metrics-certs" pod "1f0c46b1-d75f-4f7d-b10d-5899d1770e64" (UID: "1f0c46b1-d75f-4f7d-b10d-5899d1770e64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:49.649337 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.649298 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:49.649337 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.649335 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:49.649556 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.649350 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzgn\" (UniqueName: \"kubernetes.io/projected/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-kube-api-access-jrzgn\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:49.649556 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.649366 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0c46b1-d75f-4f7d-b10d-5899d1770e64-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:49.679831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.679728 4227 generic.go:356] "Generic (PLEG): container finished" podID="1f0c46b1-d75f-4f7d-b10d-5899d1770e64" containerID="de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f" exitCode=0 Nov 03 21:09:49.679831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.679795 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" Nov 03 21:09:49.680061 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.679827 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" event={"ID":"1f0c46b1-d75f-4f7d-b10d-5899d1770e64","Type":"ContainerDied","Data":"de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f"} Nov 03 21:09:49.680061 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.679881 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw" event={"ID":"1f0c46b1-d75f-4f7d-b10d-5899d1770e64","Type":"ContainerDied","Data":"dac586c84a4366a12a4ff37dee8ec631c02b7ed3dcd1bd73ee36715d84576760"} Nov 03 21:09:49.680061 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.679905 4227 scope.go:122] "RemoveContainer" containerID="de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f" Nov 03 21:09:49.691568 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.691144 4227 scope.go:122] "RemoveContainer" containerID="de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f" Nov 03 21:09:49.691772 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:49.691746 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f\": container with ID starting with de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f not found: ID does not exist" containerID="de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f" Nov 03 21:09:49.691839 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.691785 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f"} err="failed to get container status \"de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f\": rpc error: code = NotFound desc = could not find container \"de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f\": container with ID starting with de632d977187ec89c45163b225c9f18c3717fea7c38ba49e1828101e36a3571f not found: ID does not exist" Nov 03 21:09:49.835896 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.835855 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw"] Nov 03 21:09:49.838601 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:49.838577 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3ca125-74dbfccd59-7zvmw"] Nov 03 21:09:50.900502 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:50.900465 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0c46b1-d75f-4f7d-b10d-5899d1770e64" path="/var/lib/kubelet/pods/1f0c46b1-d75f-4f7d-b10d-5899d1770e64/volumes" Nov 03 21:09:51.717713 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:51.717677 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq"] Nov 03 21:09:51.761941 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:51.761905 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n"] Nov 03 21:09:56.082966 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:09:56.082925 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:09:57.785547 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:57.785515 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl"] Nov 03 21:09:57.786018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:09:57.785771 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" containerID="cri-o://6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2" gracePeriod=30 Nov 03 21:10:03.048225 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.048149 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:10:03.111810 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.111781 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-certs\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.111978 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.111818 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istiod-ca-cert\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.111978 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.111877 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-data\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.111978 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.111939 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-credential-socket\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.111978 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.111974 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-envoy\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.112237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112005 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-socket\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.112237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112047 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-token\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-token\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.112237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112086 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-podinfo\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.112237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112110 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-certs" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:03.112237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112192 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-kube-api-access-49bqq\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-kube-api-access-49bqq\") pod \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\" (UID: \"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7\") " Nov 03 21:10:03.112507 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112345 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-socket" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:03.112507 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112358 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-credential-socket" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:03.112507 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112366 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istiod-ca-cert" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:03.112757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112678 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.112757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112708 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.112757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112721 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.112757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112720 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-data" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:03.112757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.112736 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.114295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.114261 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-envoy" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:03.114396 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.114329 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-token" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:03.114778 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.114763 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-kube-api-access-49bqq" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "kube-api-access-49bqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:03.114818 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.114769 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-podinfo" pod "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" (UID: "c0fdb9d7-abfd-4c25-9f60-503543c9a0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:03.214450 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.214412 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.214450 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.214450 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.214651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.214466 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.214651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.214481 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49bqq\" (UniqueName: \"kubernetes.io/projected/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-kube-api-access-49bqq\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.214651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.214496 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:03.262535 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.262493 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr"] Nov 03 21:10:03.824689 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.824658 4227 generic.go:356] "Generic (PLEG): container finished" podID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerID="6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2" exitCode=0 Nov 03 21:10:03.824892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.824705 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" event={"ID":"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7","Type":"ContainerDied","Data":"6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2"} Nov 03 21:10:03.824892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.824718 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" Nov 03 21:10:03.824892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.824763 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl" event={"ID":"c0fdb9d7-abfd-4c25-9f60-503543c9a0a7","Type":"ContainerDied","Data":"b6ed545a23758b2b4a3b193a723013f514156bbf064b2c480bb9a5b5db18c9fe"} Nov 03 21:10:03.824892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.824793 4227 scope.go:122] "RemoveContainer" containerID="6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2" Nov 03 21:10:03.837671 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.837642 4227 scope.go:122] "RemoveContainer" containerID="7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5" Nov 03 21:10:03.847071 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.847047 4227 scope.go:122] "RemoveContainer" containerID="6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2" Nov 03 21:10:03.847449 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:03.847418 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2\": container with ID starting with 6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2 not found: ID does not exist" containerID="6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2" Nov 03 21:10:03.847526 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.847462 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2"} err="failed to get container status \"6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2\": rpc error: code = NotFound desc = could not find container \"6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2\": container with ID starting with 6410334e9632ebb629c54ae5b07be691ce41582119ac9e94afa583ff864852f2 not found: ID does not exist" Nov 03 21:10:03.847526 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.847488 4227 scope.go:122] "RemoveContainer" containerID="7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5" Nov 03 21:10:03.847770 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:03.847752 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5\": container with ID starting with 7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5 not found: ID does not exist" containerID="7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5" Nov 03 21:10:03.847812 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.847778 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5"} err="failed to get container status \"7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5\": rpc error: code = NotFound desc = could not find container \"7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5\": container with ID starting with 7adb32699fd86ede47b8daec3c5e5fea5a8714ed40a3f0115102669bbb23c2e5 not found: ID does not exist" Nov 03 21:10:03.979839 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:03.979807 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl"] Nov 03 21:10:04.026603 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:04.026569 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3ca125-openshift-default-76db89fb6f-tj7zl"] Nov 03 21:10:04.902918 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:04.902890 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" path="/var/lib/kubelet/pods/c0fdb9d7-abfd-4c25-9f60-503543c9a0a7/volumes" Nov 03 21:10:06.743708 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:06.743649 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" containerID="cri-o://739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423" gracePeriod=45 Nov 03 21:10:06.792777 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:06.792711 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" containerID="cri-o://a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f" gracePeriod=45 Nov 03 21:10:06.904309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:06.904269 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": dial tcp 10.134.0.38:8000: connect: connection refused" Nov 03 21:10:07.040791 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.040740 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Nov 03 21:10:07.147291 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.147262 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 21:10:07.306061 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.305961 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-dshm\") pod \"05139074-b3a2-4cdc-8041-32fd809250ac\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " Nov 03 21:10:07.306061 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306024 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05139074-b3a2-4cdc-8041-32fd809250ac-kube-api-access-wn7pw\" (UniqueName: \"kubernetes.io/projected/05139074-b3a2-4cdc-8041-32fd809250ac-kube-api-access-wn7pw\") pod \"05139074-b3a2-4cdc-8041-32fd809250ac\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " Nov 03 21:10:07.306389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306118 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-home\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-home\") pod \"05139074-b3a2-4cdc-8041-32fd809250ac\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " Nov 03 21:10:07.306389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306174 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-model-cache\") pod \"05139074-b3a2-4cdc-8041-32fd809250ac\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " Nov 03 21:10:07.306389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306224 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-tmp-dir\") pod \"05139074-b3a2-4cdc-8041-32fd809250ac\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " Nov 03 21:10:07.306389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306320 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05139074-b3a2-4cdc-8041-32fd809250ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/05139074-b3a2-4cdc-8041-32fd809250ac-tls-certs\") pod \"05139074-b3a2-4cdc-8041-32fd809250ac\" (UID: \"05139074-b3a2-4cdc-8041-32fd809250ac\") " Nov 03 21:10:07.307005 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306613 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-model-cache" pod "05139074-b3a2-4cdc-8041-32fd809250ac" (UID: "05139074-b3a2-4cdc-8041-32fd809250ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.307005 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306929 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-home" pod "05139074-b3a2-4cdc-8041-32fd809250ac" (UID: "05139074-b3a2-4cdc-8041-32fd809250ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.307005 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306976 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-tmp-dir" pod "05139074-b3a2-4cdc-8041-32fd809250ac" (UID: "05139074-b3a2-4cdc-8041-32fd809250ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.307005 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.306990 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.309512 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.309488 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-dshm" pod "05139074-b3a2-4cdc-8041-32fd809250ac" (UID: "05139074-b3a2-4cdc-8041-32fd809250ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.312297 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.312274 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05139074-b3a2-4cdc-8041-32fd809250ac-tls-certs" pod "05139074-b3a2-4cdc-8041-32fd809250ac" (UID: "05139074-b3a2-4cdc-8041-32fd809250ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:07.312414 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.312298 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05139074-b3a2-4cdc-8041-32fd809250ac-kube-api-access-wn7pw" pod "05139074-b3a2-4cdc-8041-32fd809250ac" (UID: "05139074-b3a2-4cdc-8041-32fd809250ac"). InnerVolumeSpecName "kube-api-access-wn7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:07.408568 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.408530 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.408568 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.408559 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7pw\" (UniqueName: \"kubernetes.io/projected/05139074-b3a2-4cdc-8041-32fd809250ac-kube-api-access-wn7pw\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.408568 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.408570 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.408791 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.408595 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05139074-b3a2-4cdc-8041-32fd809250ac-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.408791 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.408604 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05139074-b3a2-4cdc-8041-32fd809250ac-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.557576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.557492 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 21:10:07.712245 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712208 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-home\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-home\") pod \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " Nov 03 21:10:07.712423 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712280 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-dshm\") pod \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " Nov 03 21:10:07.712423 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712335 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-kube-api-access-8jwm9\" (UniqueName: \"kubernetes.io/projected/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-kube-api-access-8jwm9\") pod \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " Nov 03 21:10:07.712423 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712403 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tmp-dir\") pod \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " Nov 03 21:10:07.712613 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712470 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-model-cache\") pod \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " Nov 03 21:10:07.712668 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712648 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tls-certs\") pod \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\" (UID: \"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40\") " Nov 03 21:10:07.713323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712746 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-home" pod "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" (UID: "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.713323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712944 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tmp-dir" pod "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" (UID: "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.713323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.712957 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-model-cache" pod "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" (UID: "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.713323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.713260 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.713323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.713282 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.713323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.713298 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.715068 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.715047 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-dshm" pod "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" (UID: "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:07.715250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.715216 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-kube-api-access-8jwm9" pod "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" (UID: "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40"). InnerVolumeSpecName "kube-api-access-8jwm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:07.715444 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.715419 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tls-certs" pod "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" (UID: "b61ce70c-6d1c-41c7-8bc8-988ad23a3d40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:07.815096 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.815019 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.815096 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.815049 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.815096 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.815063 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwm9\" (UniqueName: \"kubernetes.io/projected/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40-kube-api-access-8jwm9\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.908861 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.908824 4227 generic.go:356] "Generic (PLEG): container finished" podID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerID="a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f" exitCode=0 Nov 03 21:10:07.909018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.908869 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" Nov 03 21:10:07.909018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.908900 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" event={"ID":"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40","Type":"ContainerDied","Data":"a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f"} Nov 03 21:10:07.909018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.908939 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n" event={"ID":"b61ce70c-6d1c-41c7-8bc8-988ad23a3d40","Type":"ContainerDied","Data":"2cbcbe761bfda585109ce220c59bc220af4a0b6e50c238e72bfc12cee57aec66"} Nov 03 21:10:07.909018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.908962 4227 scope.go:122] "RemoveContainer" containerID="a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f" Nov 03 21:10:07.910461 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.910441 4227 generic.go:356] "Generic (PLEG): container finished" podID="05139074-b3a2-4cdc-8041-32fd809250ac" containerID="739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423" exitCode=0 Nov 03 21:10:07.910546 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.910496 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" Nov 03 21:10:07.910546 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.910511 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" event={"ID":"05139074-b3a2-4cdc-8041-32fd809250ac","Type":"ContainerDied","Data":"739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423"} Nov 03 21:10:07.910546 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.910535 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq" event={"ID":"05139074-b3a2-4cdc-8041-32fd809250ac","Type":"ContainerDied","Data":"48381f79a1e81ae45f884be580f5feba2a2b7903730cc21b4f18caef1414a688"} Nov 03 21:10:07.920093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.920075 4227 scope.go:122] "RemoveContainer" containerID="a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f" Nov 03 21:10:07.920548 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:07.920469 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f\": container with ID starting with a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f not found: ID does not exist" containerID="a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f" Nov 03 21:10:07.920548 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.920510 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f"} err="failed to get container status \"a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f\": rpc error: code = NotFound desc = could not find container \"a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f\": container with ID starting with a322ce31724c72d1e174bf5cf7ab409738c69a2a34ffeebb54f42d9db330249f not found: ID does not exist" Nov 03 21:10:07.920548 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.920534 4227 scope.go:122] "RemoveContainer" containerID="739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423" Nov 03 21:10:07.928862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.928843 4227 scope.go:122] "RemoveContainer" containerID="739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423" Nov 03 21:10:07.929148 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:07.929130 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423\": container with ID starting with 739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423 not found: ID does not exist" containerID="739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423" Nov 03 21:10:07.929222 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:07.929153 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423"} err="failed to get container status \"739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423\": rpc error: code = NotFound desc = could not find container \"739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423\": container with ID starting with 739a223de3bd8ea50c2dc4ab3defc7479c7f05a354bef2477a50262d70f41423 not found: ID does not exist" Nov 03 21:10:08.062521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:08.062491 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq"] Nov 03 21:10:08.109330 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:08.109262 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/distinct-2-w0-fd9afe-kserve-67964f5fc5-xd8hq"] Nov 03 21:10:08.212003 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:08.211968 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n"] Nov 03 21:10:08.215442 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:08.215417 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/premium-w0-fd9afe-kserve-67dfcd684f-gll8n"] Nov 03 21:10:08.903490 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:08.903450 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" path="/var/lib/kubelet/pods/05139074-b3a2-4cdc-8041-32fd809250ac/volumes" Nov 03 21:10:08.904744 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:08.904717 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" path="/var/lib/kubelet/pods/b61ce70c-6d1c-41c7-8bc8-988ad23a3d40/volumes" Nov 03 21:10:12.627597 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:12.627557 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t"] Nov 03 21:10:12.628677 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:12.628329 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" containerID="cri-o://5666b8c36626bc912f50ffb98541e4d3bef331d6c5cdeb19faf75b719c73f9fb" gracePeriod=30 Nov 03 21:10:12.652582 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:12.651662 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7"] Nov 03 21:10:12.652981 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:12.652947 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" containerID="cri-o://c447ff2fa5769aa0be63dcc70aa0f4ca8e403d0eff582b11ebe90c691ab96699" gracePeriod=30 Nov 03 21:10:12.669414 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:12.669387 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz"] Nov 03 21:10:12.669638 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:12.669615 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" containerID="cri-o://e1ff6da966cd6322b739b717750e60a5573e63024e28faac63a752337380a21c" gracePeriod=30 Nov 03 21:10:15.217824 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:15.217787 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2"] Nov 03 21:10:15.218599 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:15.218049 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" containerID="cri-o://837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c" gracePeriod=30 Nov 03 21:10:17.100200 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:17.100138 4227 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:10:17.100605 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:17.100238 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:10:18.012684 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.012651 4227 generic.go:356] "Generic (PLEG): container finished" podID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerID="5666b8c36626bc912f50ffb98541e4d3bef331d6c5cdeb19faf75b719c73f9fb" exitCode=0 Nov 03 21:10:18.013102 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.012918 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" event={"ID":"c494a8ed-803e-4b5c-8854-960dac3d13fd","Type":"ContainerDied","Data":"5666b8c36626bc912f50ffb98541e4d3bef331d6c5cdeb19faf75b719c73f9fb"} Nov 03 21:10:18.017820 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.017559 4227 generic.go:356] "Generic (PLEG): container finished" podID="71d08ec1-f534-45c2-933b-410a57767c15" containerID="c447ff2fa5769aa0be63dcc70aa0f4ca8e403d0eff582b11ebe90c691ab96699" exitCode=0 Nov 03 21:10:18.017820 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.017755 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" event={"ID":"71d08ec1-f534-45c2-933b-410a57767c15","Type":"ContainerDied","Data":"c447ff2fa5769aa0be63dcc70aa0f4ca8e403d0eff582b11ebe90c691ab96699"} Nov 03 21:10:18.021303 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.021212 4227 generic.go:356] "Generic (PLEG): container finished" podID="f9b1bba5-799f-451b-859c-741433af9d3c" containerID="e1ff6da966cd6322b739b717750e60a5573e63024e28faac63a752337380a21c" exitCode=0 Nov 03 21:10:18.021303 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.021254 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" event={"ID":"f9b1bba5-799f-451b-859c-741433af9d3c","Type":"ContainerDied","Data":"e1ff6da966cd6322b739b717750e60a5573e63024e28faac63a752337380a21c"} Nov 03 21:10:18.050550 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.050516 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:10:18.097845 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.097817 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:10:18.166814 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.166782 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-kube-api-access-zfjmt\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-kube-api-access-zfjmt\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.167208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.166907 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d08ec1-f534-45c2-933b-410a57767c15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d08ec1-f534-45c2-933b-410a57767c15-istio-podinfo\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.167208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.166990 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-istio-token\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-istio-token\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.167386 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.167365 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-socket\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.167462 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.167444 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-data\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.167521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.167482 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d08ec1-f534-45c2-933b-410a57767c15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d08ec1-f534-45c2-933b-410a57767c15-istiod-ca-cert\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.167518 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-credential-socket\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.167593 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-envoy\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.167634 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-certs\") pod \"71d08ec1-f534-45c2-933b-410a57767c15\" (UID: \"71d08ec1-f534-45c2-933b-410a57767c15\") " Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.167968 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-credential-socket" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.168339 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-socket" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.168627 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.168642 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.168688 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-data" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.168727 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-certs" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.168893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.168848 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d08ec1-f534-45c2-933b-410a57767c15-istiod-ca-cert" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:18.169375 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.169308 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d08ec1-f534-45c2-933b-410a57767c15-istio-podinfo" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:18.169423 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.169357 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-kube-api-access-zfjmt" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "kube-api-access-zfjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.169774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.169747 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-istio-token" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.170987 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.170968 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-envoy" pod "71d08ec1-f534-45c2-933b-410a57767c15" (UID: "71d08ec1-f534-45c2-933b-410a57767c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.270106 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.270049 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-data\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.270318 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.270207 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-kube-api-access-zcnrv\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-kube-api-access-zcnrv\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.270318 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.270244 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-envoy\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.270318 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.270300 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-istio-token\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.270492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.270400 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-certs\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.270651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.270546 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-socket\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.270651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.270584 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b1bba5-799f-451b-859c-741433af9d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b1bba5-799f-451b-859c-741433af9d3c-istiod-ca-cert\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.271330 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.271303 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-data" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.271960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.271802 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9b1bba5-799f-451b-859c-741433af9d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b1bba5-799f-451b-859c-741433af9d3c-istio-podinfo\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.271960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.271856 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-credential-socket\") pod \"f9b1bba5-799f-451b-859c-741433af9d3c\" (UID: \"f9b1bba5-799f-451b-859c-741433af9d3c\") " Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272514 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272539 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d08ec1-f534-45c2-933b-410a57767c15-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272549 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272558 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d08ec1-f534-45c2-933b-410a57767c15-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272567 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272578 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfjmt\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-kube-api-access-zfjmt\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272586 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d08ec1-f534-45c2-933b-410a57767c15-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272595 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d08ec1-f534-45c2-933b-410a57767c15-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.272774 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-credential-socket" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.273294 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b1bba5-799f-451b-859c-741433af9d3c-istiod-ca-cert" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.273834 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-certs" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.274014 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-socket" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.275729 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-envoy" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.276375 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-kube-api-access-zcnrv" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "kube-api-access-zcnrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.277274 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-istio-token" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.280202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.277420 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9b1bba5-799f-451b-859c-741433af9d3c-istio-podinfo" pod "f9b1bba5-799f-451b-859c-741433af9d3c" (UID: "f9b1bba5-799f-451b-859c-741433af9d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:18.283197 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.283152 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" containerID="cri-o://e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592" gracePeriod=45 Nov 03 21:10:18.354150 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.354124 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:10:18.373894 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373868 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b1bba5-799f-451b-859c-741433af9d3c-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.373894 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373892 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.373894 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373905 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnrv\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-kube-api-access-zcnrv\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.374210 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373917 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.374210 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373931 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b1bba5-799f-451b-859c-741433af9d3c-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.374210 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373943 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.374210 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373954 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b1bba5-799f-451b-859c-741433af9d3c-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.374210 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.373962 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b1bba5-799f-451b-859c-741433af9d3c-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505532 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c494a8ed-803e-4b5c-8854-960dac3d13fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c494a8ed-803e-4b5c-8854-960dac3d13fd-istiod-ca-cert\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505655 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-data\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505710 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-socket\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505743 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-podinfo\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505777 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-kube-api-access-llcx7\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-kube-api-access-llcx7\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505812 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-envoy\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505916 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-credential-socket\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505945 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-token\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-token\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.506365 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.505990 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-certs\") pod \"c494a8ed-803e-4b5c-8854-960dac3d13fd\" (UID: \"c494a8ed-803e-4b5c-8854-960dac3d13fd\") " Nov 03 21:10:18.521852 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.510657 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-credential-socket" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.521852 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.510953 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c494a8ed-803e-4b5c-8854-960dac3d13fd-istiod-ca-cert" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:18.521852 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.511778 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-data" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.521852 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.512788 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-socket" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.521852 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.519991 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-certs" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.534708 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.534679 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-token" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.551025 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.544526 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-podinfo" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:18.551025 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.548465 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-envoy" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.565742 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.565215 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-kube-api-access-llcx7" pod "c494a8ed-803e-4b5c-8854-960dac3d13fd" (UID: "c494a8ed-803e-4b5c-8854-960dac3d13fd"). InnerVolumeSpecName "kube-api-access-llcx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617142 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617198 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617213 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617227 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c494a8ed-803e-4b5c-8854-960dac3d13fd-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617243 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617256 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617270 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617283 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llcx7\" (UniqueName: \"kubernetes.io/projected/c494a8ed-803e-4b5c-8854-960dac3d13fd-kube-api-access-llcx7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.623208 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.617297 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c494a8ed-803e-4b5c-8854-960dac3d13fd-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.810571 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.810545 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 21:10:18.921709 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.921680 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-home\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-home\") pod \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " Nov 03 21:10:18.921844 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.921746 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-model-cache\") pod \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " Nov 03 21:10:18.921909 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.921850 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tls-certs\") pod \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " Nov 03 21:10:18.921968 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.921913 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-kube-api-access-26t6f\" (UniqueName: \"kubernetes.io/projected/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-kube-api-access-26t6f\") pod \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " Nov 03 21:10:18.921968 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.921947 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-dshm\") pod \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " Nov 03 21:10:18.922069 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.921963 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-home" pod "7a61d61f-f982-4f59-adaa-c0c7210bb7d3" (UID: "7a61d61f-f982-4f59-adaa-c0c7210bb7d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.922069 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.922011 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tmp-dir\") pod \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\" (UID: \"7a61d61f-f982-4f59-adaa-c0c7210bb7d3\") " Nov 03 21:10:18.922191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.922099 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-model-cache" pod "7a61d61f-f982-4f59-adaa-c0c7210bb7d3" (UID: "7a61d61f-f982-4f59-adaa-c0c7210bb7d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.922390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.922369 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tmp-dir" pod "7a61d61f-f982-4f59-adaa-c0c7210bb7d3" (UID: "7a61d61f-f982-4f59-adaa-c0c7210bb7d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.922549 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.922434 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.922549 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.922453 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:18.936623 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.936575 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-dshm" pod "7a61d61f-f982-4f59-adaa-c0c7210bb7d3" (UID: "7a61d61f-f982-4f59-adaa-c0c7210bb7d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:18.948530 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.948498 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tls-certs" pod "7a61d61f-f982-4f59-adaa-c0c7210bb7d3" (UID: "7a61d61f-f982-4f59-adaa-c0c7210bb7d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:18.948634 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:18.948504 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-kube-api-access-26t6f" pod "7a61d61f-f982-4f59-adaa-c0c7210bb7d3" (UID: "7a61d61f-f982-4f59-adaa-c0c7210bb7d3"). InnerVolumeSpecName "kube-api-access-26t6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:19.023647 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.023615 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:19.023647 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.023649 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26t6f\" (UniqueName: \"kubernetes.io/projected/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-kube-api-access-26t6f\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:19.029437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.023664 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:19.029437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.023679 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a61d61f-f982-4f59-adaa-c0c7210bb7d3-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:19.034045 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.033972 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" Nov 03 21:10:19.034045 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.033986 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7" event={"ID":"71d08ec1-f534-45c2-933b-410a57767c15","Type":"ContainerDied","Data":"74ce720efbc9a38dcaae47cbeef77d98dd648702b04c7dee803b527602068d83"} Nov 03 21:10:19.034258 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.034087 4227 scope.go:122] "RemoveContainer" containerID="c447ff2fa5769aa0be63dcc70aa0f4ca8e403d0eff582b11ebe90c691ab96699" Nov 03 21:10:19.035580 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.035556 4227 generic.go:356] "Generic (PLEG): container finished" podID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerID="e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592" exitCode=0 Nov 03 21:10:19.035689 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.035631 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" Nov 03 21:10:19.035689 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.035637 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" event={"ID":"7a61d61f-f982-4f59-adaa-c0c7210bb7d3","Type":"ContainerDied","Data":"e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592"} Nov 03 21:10:19.035689 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.035673 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" event={"ID":"7a61d61f-f982-4f59-adaa-c0c7210bb7d3","Type":"ContainerDied","Data":"e76d32f187410ae3916b1b2b2d96a7f21a488ae84da50167c7243cbf962ad4d1"} Nov 03 21:10:19.037449 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.037426 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" event={"ID":"f9b1bba5-799f-451b-859c-741433af9d3c","Type":"ContainerDied","Data":"955337c1611bcc058e80ed81b63d47ae4f26cdfaade356bd59c3b4c000876734"} Nov 03 21:10:19.037541 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.037497 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz" Nov 03 21:10:19.038664 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.038641 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" event={"ID":"c494a8ed-803e-4b5c-8854-960dac3d13fd","Type":"ContainerDied","Data":"93667488a62e180b3ebb117e58e284d752bb00568b1cae695d6cebd64beb4ebb"} Nov 03 21:10:19.038764 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.038684 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t" Nov 03 21:10:19.045776 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.045741 4227 scope.go:122] "RemoveContainer" containerID="e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592" Nov 03 21:10:19.053788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.053769 4227 scope.go:122] "RemoveContainer" containerID="e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592" Nov 03 21:10:19.054044 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:19.054026 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592\": container with ID starting with e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592 not found: ID does not exist" containerID="e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592" Nov 03 21:10:19.054123 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.054057 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592"} err="failed to get container status \"e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592\": rpc error: code = NotFound desc = could not find container \"e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592\": container with ID starting with e9904007a1eab3ee12779adda58811f24d110c06fcf67859eca01d9ade1fc592 not found: ID does not exist" Nov 03 21:10:19.054123 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.054080 4227 scope.go:122] "RemoveContainer" containerID="e1ff6da966cd6322b739b717750e60a5573e63024e28faac63a752337380a21c" Nov 03 21:10:19.065294 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.065278 4227 scope.go:122] "RemoveContainer" containerID="5666b8c36626bc912f50ffb98541e4d3bef331d6c5cdeb19faf75b719c73f9fb" Nov 03 21:10:19.193583 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.193546 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t"] Nov 03 21:10:19.198427 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.198401 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-7kf9t"] Nov 03 21:10:19.301590 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.301518 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz"] Nov 03 21:10:19.310575 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.310544 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-m4nlz"] Nov 03 21:10:19.416720 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.416690 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr"] Nov 03 21:10:19.419429 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.419401 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr"] Nov 03 21:10:19.526430 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.526395 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7"] Nov 03 21:10:19.528704 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.528678 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-sdrv7"] Nov 03 21:10:19.727069 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:19.727029 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-w2-b119f2-kserve-6dff96484d-ssfxr" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:10:20.536334 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.536312 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 21:10:20.641859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.641821 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-envoy\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642023 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.641910 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-kube-api-access-vqh9p\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-kube-api-access-vqh9p\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642018 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-certs\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642049 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-data\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642086 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-podinfo\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642121 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istiod-ca-cert\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642187 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-credential-socket\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642227 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-socket\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642264 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-token\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-token\") pod \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\" (UID: \"968f7168-1fe9-4ff6-a5a3-3d4e16d97207\") " Nov 03 21:10:20.642492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642335 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-certs" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:20.642584 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642562 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-credential-socket" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:20.642751 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642726 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-socket" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:20.642839 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642814 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istiod-ca-cert" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:20.642901 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642884 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.642956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642906 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.642956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.642922 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.643218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.643131 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-data" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:20.644615 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.644581 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-kube-api-access-vqh9p" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "kube-api-access-vqh9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:20.644724 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.644668 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-envoy" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:20.644948 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.644925 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-podinfo" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:20.645074 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.645057 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-token" pod "968f7168-1fe9-4ff6-a5a3-3d4e16d97207" (UID: "968f7168-1fe9-4ff6-a5a3-3d4e16d97207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:20.744020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.743988 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.744020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.744017 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqh9p\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-kube-api-access-vqh9p\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.744020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.744026 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.744259 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.744035 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.744259 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.744043 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.744259 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.744051 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968f7168-1fe9-4ff6-a5a3-3d4e16d97207-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:20.899391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.899365 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d08ec1-f534-45c2-933b-410a57767c15" path="/var/lib/kubelet/pods/71d08ec1-f534-45c2-933b-410a57767c15/volumes" Nov 03 21:10:20.899819 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.899806 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" path="/var/lib/kubelet/pods/7a61d61f-f982-4f59-adaa-c0c7210bb7d3/volumes" Nov 03 21:10:20.900403 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.900377 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" path="/var/lib/kubelet/pods/c494a8ed-803e-4b5c-8854-960dac3d13fd/volumes" Nov 03 21:10:20.901106 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:20.901085 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" path="/var/lib/kubelet/pods/f9b1bba5-799f-451b-859c-741433af9d3c/volumes" Nov 03 21:10:21.082120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.082086 4227 generic.go:356] "Generic (PLEG): container finished" podID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerID="837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c" exitCode=0 Nov 03 21:10:21.082299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.082135 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" Nov 03 21:10:21.082299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.082184 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" event={"ID":"968f7168-1fe9-4ff6-a5a3-3d4e16d97207","Type":"ContainerDied","Data":"837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c"} Nov 03 21:10:21.082299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.082249 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2" event={"ID":"968f7168-1fe9-4ff6-a5a3-3d4e16d97207","Type":"ContainerDied","Data":"708007e531682b0936a8a5284af77ea77eaef6f06e86c2ed49e29d5d16597d62"} Nov 03 21:10:21.082299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.082277 4227 scope.go:122] "RemoveContainer" containerID="837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c" Nov 03 21:10:21.091116 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.091096 4227 scope.go:122] "RemoveContainer" containerID="837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c" Nov 03 21:10:21.091423 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:21.091398 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c\": container with ID starting with 837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c not found: ID does not exist" containerID="837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c" Nov 03 21:10:21.091478 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.091434 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c"} err="failed to get container status \"837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c\": rpc error: code = NotFound desc = could not find container \"837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c\": container with ID starting with 837aa468770c77c8ee5ecd5629623231ec1cad8df68853be47570466aaf4f30c not found: ID does not exist" Nov 03 21:10:21.235183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.235065 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2"] Nov 03 21:10:21.281361 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:21.281322 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-9kht2"] Nov 03 21:10:22.900217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:22.900185 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" path="/var/lib/kubelet/pods/968f7168-1fe9-4ff6-a5a3-3d4e16d97207/volumes" Nov 03 21:10:26.087629 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:26.087582 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:10:29.132642 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:29.132605 4227 generic.go:356] "Generic (PLEG): container finished" podID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerID="b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5" exitCode=0 Nov 03 21:10:29.133090 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:29.132678 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" event={"ID":"f6ca7467-f49f-4151-85fc-5fc490e9a518","Type":"ContainerDied","Data":"b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5"} Nov 03 21:10:29.133090 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:29.133049 4227 scope.go:122] "RemoveContainer" containerID="b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5" Nov 03 21:10:29.134403 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:29.134377 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:29.134474 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:29.134442 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:29.134474 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:29.134471 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:30.139892 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:30.139852 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" event={"ID":"f6ca7467-f49f-4151-85fc-5fc490e9a518","Type":"ContainerStarted","Data":"01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c"} Nov 03 21:10:30.625732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:30.625696 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:10:30.626468 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:30.626444 4227 patch_prober.go:29] interesting pod/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:10:30.626545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:30.626503 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:10:31.031487 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:31.031457 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:10:31.628370 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:31.628332 4227 patch_prober.go:29] interesting pod/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:10:31.629000 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:31.628965 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:10:32.625806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:32.625769 4227 patch_prober.go:29] interesting pod/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:10:32.625993 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:32.625822 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:10:33.626336 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:33.626298 4227 patch_prober.go:29] interesting pod/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:10:33.626726 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:33.626357 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:10:34.625552 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:34.625517 4227 patch_prober.go:29] interesting pod/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:10:34.625759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:34.625574 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:10:35.779461 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:35.779431 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:10:35.781492 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:35.781453 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:10:36.813921 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:36.813885 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk"] Nov 03 21:10:36.814364 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:36.814096 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" podUID="2f7bdc91-c5c7-4abd-9993-815c3623b258" containerName="maas-api" containerID="cri-o://77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0" gracePeriod=30 Nov 03 21:10:37.121139 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.121111 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 21:10:37.240626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.239429 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls\") pod \"2f7bdc91-c5c7-4abd-9993-815c3623b258\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " Nov 03 21:10:37.240626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.239496 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7bdc91-c5c7-4abd-9993-815c3623b258-kube-api-access-xk2pc\" (UniqueName: \"kubernetes.io/projected/2f7bdc91-c5c7-4abd-9993-815c3623b258-kube-api-access-xk2pc\") pod \"2f7bdc91-c5c7-4abd-9993-815c3623b258\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " Nov 03 21:10:37.240626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.239560 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7bdc91-c5c7-4abd-9993-815c3623b258-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7bdc91-c5c7-4abd-9993-815c3623b258-openshift-service-ca\") pod \"2f7bdc91-c5c7-4abd-9993-815c3623b258\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " Nov 03 21:10:37.240626 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.239694 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-metrics-certs\") pod \"2f7bdc91-c5c7-4abd-9993-815c3623b258\" (UID: \"2f7bdc91-c5c7-4abd-9993-815c3623b258\") " Nov 03 21:10:37.243210 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.243132 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7bdc91-c5c7-4abd-9993-815c3623b258-openshift-service-ca" pod "2f7bdc91-c5c7-4abd-9993-815c3623b258" (UID: "2f7bdc91-c5c7-4abd-9993-815c3623b258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:37.270956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.268445 4227 generic.go:356] "Generic (PLEG): container finished" podID="2f7bdc91-c5c7-4abd-9993-815c3623b258" containerID="77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0" exitCode=0 Nov 03 21:10:37.270956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.268555 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" event={"ID":"2f7bdc91-c5c7-4abd-9993-815c3623b258","Type":"ContainerDied","Data":"77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0"} Nov 03 21:10:37.270956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.268606 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" event={"ID":"2f7bdc91-c5c7-4abd-9993-815c3623b258","Type":"ContainerDied","Data":"18460c20b6f5d1b6511ac2e84a89ffce0ea2ad6981ac7af0edf6e7bbdfe837f1"} Nov 03 21:10:37.270956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.268642 4227 scope.go:122] "RemoveContainer" containerID="77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0" Nov 03 21:10:37.270956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.268860 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk" Nov 03 21:10:37.279369 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.279301 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls" pod "2f7bdc91-c5c7-4abd-9993-815c3623b258" (UID: "2f7bdc91-c5c7-4abd-9993-815c3623b258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:37.279689 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.279647 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7bdc91-c5c7-4abd-9993-815c3623b258-kube-api-access-xk2pc" pod "2f7bdc91-c5c7-4abd-9993-815c3623b258" (UID: "2f7bdc91-c5c7-4abd-9993-815c3623b258"). InnerVolumeSpecName "kube-api-access-xk2pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:37.281000 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.280946 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-metrics-certs" pod "2f7bdc91-c5c7-4abd-9993-815c3623b258" (UID: "2f7bdc91-c5c7-4abd-9993-815c3623b258"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:37.299437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.299414 4227 scope.go:122] "RemoveContainer" containerID="77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0" Nov 03 21:10:37.299839 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:37.299807 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0\": container with ID starting with 77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0 not found: ID does not exist" containerID="77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0" Nov 03 21:10:37.299960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.299842 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0"} err="failed to get container status \"77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0\": rpc error: code = NotFound desc = could not find container \"77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0\": container with ID starting with 77e0c6edf378e7dfbbb8700be4467ec7d8f733d1e011a954ed8c3a84f74ea1f0 not found: ID does not exist" Nov 03 21:10:37.341484 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.341404 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:37.341484 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.341432 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2pc\" (UniqueName: \"kubernetes.io/projected/2f7bdc91-c5c7-4abd-9993-815c3623b258-kube-api-access-xk2pc\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:37.341484 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.341445 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7bdc91-c5c7-4abd-9993-815c3623b258-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:37.341484 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.341457 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7bdc91-c5c7-4abd-9993-815c3623b258-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:37.682118 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.682090 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk"] Nov 03 21:10:37.773298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:37.771920 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1ec892-58cc7477fd-dmkjk"] Nov 03 21:10:38.921562 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:38.921525 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7bdc91-c5c7-4abd-9993-815c3623b258" path="/var/lib/kubelet/pods/2f7bdc91-c5c7-4abd-9993-815c3623b258/volumes" Nov 03 21:10:45.632887 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:45.632851 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4"] Nov 03 21:10:45.633420 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:45.633113 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" containerID="cri-o://3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581" gracePeriod=30 Nov 03 21:10:48.299415 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:48.299381 4227 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:10:48.299852 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:48.299449 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:10:50.921775 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:50.921750 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 21:10:51.008474 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008441 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-data\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.008636 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008531 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-kube-api-access-7sdsm\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-kube-api-access-7sdsm\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.008686 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008666 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-istio-token\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-istio-token\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.008733 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008711 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-credential-socket\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.008788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008745 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36240b08-909a-47c5-a475-8d8fac2fb203-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36240b08-909a-47c5-a475-8d8fac2fb203-istio-podinfo\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.008788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008750 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-data" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:51.008906 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008818 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-certs\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.008906 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008856 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-envoy\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.008906 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008890 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-socket\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.009060 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008927 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36240b08-909a-47c5-a475-8d8fac2fb203-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36240b08-909a-47c5-a475-8d8fac2fb203-istiod-ca-cert\") pod \"36240b08-909a-47c5-a475-8d8fac2fb203\" (UID: \"36240b08-909a-47c5-a475-8d8fac2fb203\") " Nov 03 21:10:51.009060 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.008967 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-credential-socket" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:51.009225 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.009202 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-socket" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:51.009579 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.009484 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.009579 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.009496 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36240b08-909a-47c5-a475-8d8fac2fb203-istiod-ca-cert" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:51.009579 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.009510 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.009579 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.009514 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-certs" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:51.009579 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.009527 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.011020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.010997 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36240b08-909a-47c5-a475-8d8fac2fb203-istio-podinfo" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:51.011276 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.011251 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-envoy" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:51.011368 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.011262 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-kube-api-access-7sdsm" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "kube-api-access-7sdsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:51.011510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.011494 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-istio-token" pod "36240b08-909a-47c5-a475-8d8fac2fb203" (UID: "36240b08-909a-47c5-a475-8d8fac2fb203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:51.111144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.111102 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.111144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.111138 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36240b08-909a-47c5-a475-8d8fac2fb203-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.111413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.111169 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sdsm\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-kube-api-access-7sdsm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.111413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.111188 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36240b08-909a-47c5-a475-8d8fac2fb203-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.111413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.111201 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36240b08-909a-47c5-a475-8d8fac2fb203-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.111413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.111214 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36240b08-909a-47c5-a475-8d8fac2fb203-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:51.351266 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.351146 4227 generic.go:356] "Generic (PLEG): container finished" podID="36240b08-909a-47c5-a475-8d8fac2fb203" containerID="3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581" exitCode=0 Nov 03 21:10:51.351266 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.351219 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" Nov 03 21:10:51.351266 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.351223 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" event={"ID":"36240b08-909a-47c5-a475-8d8fac2fb203","Type":"ContainerDied","Data":"3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581"} Nov 03 21:10:51.351551 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.351293 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4" event={"ID":"36240b08-909a-47c5-a475-8d8fac2fb203","Type":"ContainerDied","Data":"f29c12455a889e75f61011b6086d0b0f38f8c454b1a5b292ac96c876c6a8b252"} Nov 03 21:10:51.351551 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.351325 4227 scope.go:122] "RemoveContainer" containerID="3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581" Nov 03 21:10:51.362108 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.362089 4227 scope.go:122] "RemoveContainer" containerID="3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581" Nov 03 21:10:51.362511 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:51.362487 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581\": container with ID starting with 3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581 not found: ID does not exist" containerID="3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581" Nov 03 21:10:51.362579 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.362525 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581"} err="failed to get container status \"3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581\": rpc error: code = NotFound desc = could not find container \"3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581\": container with ID starting with 3d8cdf9efb6c3fae0c737767b10f21696842240224323b87cdfae07619c14581 not found: ID does not exist" Nov 03 21:10:51.504406 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.504373 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4"] Nov 03 21:10:51.551914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:51.551880 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-6b4q4"] Nov 03 21:10:52.359853 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:52.359828 4227 generic.go:356] "Generic (PLEG): container finished" podID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerID="7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f" exitCode=0 Nov 03 21:10:52.360237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:52.359902 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" event={"ID":"f6278553-4b48-4f8c-bbc3-d3217718265c","Type":"ContainerDied","Data":"7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f"} Nov 03 21:10:52.360412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:52.360393 4227 scope.go:122] "RemoveContainer" containerID="7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f" Nov 03 21:10:52.362020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:52.361993 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:52.362103 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:52.362067 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:52.362144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:52.362103 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:52.900243 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:52.900213 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" path="/var/lib/kubelet/pods/36240b08-909a-47c5-a475-8d8fac2fb203/volumes" Nov 03 21:10:53.378419 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:53.378319 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" event={"ID":"f6278553-4b48-4f8c-bbc3-d3217718265c","Type":"ContainerStarted","Data":"302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0"} Nov 03 21:10:53.614204 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:53.614147 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:10:53.614479 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:53.614458 4227 patch_prober.go:29] interesting pod/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:10:53.614538 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:53.614503 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 21:10:54.614779 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:54.614740 4227 patch_prober.go:29] interesting pod/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:10:54.615200 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:54.614799 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 21:10:55.614859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:55.614825 4227 patch_prober.go:29] interesting pod/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:10:55.615285 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:55.614888 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 21:10:56.104495 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:10:56.104392 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:10:56.530425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.530390 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk"] Nov 03 21:10:56.531284 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.531259 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" Nov 03 21:10:56.531444 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.531429 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" Nov 03 21:10:56.531550 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.531539 4227 cpu_manager.go:414] "containerMap: removing container" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" Nov 03 21:10:56.531665 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.531651 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" Nov 03 21:10:56.531777 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.531765 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" Nov 03 21:10:56.531862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.531851 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" Nov 03 21:10:56.531964 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.531953 4227 cpu_manager.go:414] "containerMap: removing container" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" Nov 03 21:10:56.532053 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532043 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" Nov 03 21:10:56.532136 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532127 4227 cpu_manager.go:414] "containerMap: removing container" podUID="b8ce8e3d-bf70-46a2-b705-f52e42403c79" containerName="maas-api" Nov 03 21:10:56.532245 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532232 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ce8e3d-bf70-46a2-b705-f52e42403c79" containerName="maas-api" Nov 03 21:10:56.532337 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532327 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" Nov 03 21:10:56.532418 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532408 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" Nov 03 21:10:56.532518 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532506 4227 cpu_manager.go:414] "containerMap: removing container" podUID="0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" containerName="payload-processing" Nov 03 21:10:56.532604 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532593 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" containerName="payload-processing" Nov 03 21:10:56.532687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532676 4227 cpu_manager.go:414] "containerMap: removing container" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" Nov 03 21:10:56.532761 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532751 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" Nov 03 21:10:56.532858 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532849 4227 cpu_manager.go:414] "containerMap: removing container" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" Nov 03 21:10:56.532934 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.532923 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" Nov 03 21:10:56.533024 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533014 4227 cpu_manager.go:414] "containerMap: removing container" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" Nov 03 21:10:56.533106 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533096 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" Nov 03 21:10:56.533211 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533199 4227 cpu_manager.go:414] "containerMap: removing container" podUID="7e32d66d-1706-4c7f-8465-727dc51387f9" containerName="main" Nov 03 21:10:56.533301 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533290 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e32d66d-1706-4c7f-8465-727dc51387f9" containerName="main" Nov 03 21:10:56.533397 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533384 4227 cpu_manager.go:414] "containerMap: removing container" podUID="2c642461-3a57-4cca-89d1-20caf9f69d44" containerName="payload-pre-processing" Nov 03 21:10:56.533485 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533472 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c642461-3a57-4cca-89d1-20caf9f69d44" containerName="payload-pre-processing" Nov 03 21:10:56.533576 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533565 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" Nov 03 21:10:56.533658 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533647 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" Nov 03 21:10:56.533748 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533738 4227 cpu_manager.go:414] "containerMap: removing container" podUID="2f7bdc91-c5c7-4abd-9993-815c3623b258" containerName="maas-api" Nov 03 21:10:56.533832 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533820 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7bdc91-c5c7-4abd-9993-815c3623b258" containerName="maas-api" Nov 03 21:10:56.533938 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.533928 4227 cpu_manager.go:414] "containerMap: removing container" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" Nov 03 21:10:56.534024 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534013 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" Nov 03 21:10:56.534114 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534104 4227 cpu_manager.go:414] "containerMap: removing container" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" Nov 03 21:10:56.534215 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534203 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" Nov 03 21:10:56.534306 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534296 4227 cpu_manager.go:414] "containerMap: removing container" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" Nov 03 21:10:56.534388 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534377 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" Nov 03 21:10:56.534481 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534471 4227 cpu_manager.go:414] "containerMap: removing container" podUID="1f0c46b1-d75f-4f7d-b10d-5899d1770e64" containerName="maas-api" Nov 03 21:10:56.534561 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534551 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0c46b1-d75f-4f7d-b10d-5899d1770e64" containerName="maas-api" Nov 03 21:10:56.534751 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534736 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b1bba5-799f-451b-859c-741433af9d3c" containerName="istio-proxy" Nov 03 21:10:56.534857 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534846 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="36240b08-909a-47c5-a475-8d8fac2fb203" containerName="istio-proxy" Nov 03 21:10:56.534941 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.534931 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" Nov 03 21:10:56.535021 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535011 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec8860f-fba0-4a16-b9c6-c155a51f9a2c" containerName="payload-processing" Nov 03 21:10:56.535103 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535093 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff579ec-c609-4092-b348-6f4c24e2dc93" containerName="main" Nov 03 21:10:56.535214 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535202 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ce8e3d-bf70-46a2-b705-f52e42403c79" containerName="maas-api" Nov 03 21:10:56.535310 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535298 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ad67a2-6719-4c18-9f54-fc3633bccc78" containerName="main" Nov 03 21:10:56.535405 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535394 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a61d61f-f982-4f59-adaa-c0c7210bb7d3" containerName="main" Nov 03 21:10:56.535486 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535476 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="968f7168-1fe9-4ff6-a5a3-3d4e16d97207" containerName="istio-proxy" Nov 03 21:10:56.535565 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535555 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e32d66d-1706-4c7f-8465-727dc51387f9" containerName="main" Nov 03 21:10:56.535646 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535636 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c642461-3a57-4cca-89d1-20caf9f69d44" containerName="payload-pre-processing" Nov 03 21:10:56.535727 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535718 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c494a8ed-803e-4b5c-8854-960dac3d13fd" containerName="istio-proxy" Nov 03 21:10:56.535808 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535798 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7bdc91-c5c7-4abd-9993-815c3623b258" containerName="maas-api" Nov 03 21:10:56.535895 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535885 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61ce70c-6d1c-41c7-8bc8-988ad23a3d40" containerName="main" Nov 03 21:10:56.535974 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.535965 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="05139074-b3a2-4cdc-8041-32fd809250ac" containerName="main" Nov 03 21:10:56.536052 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.536043 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d08ec1-f534-45c2-933b-410a57767c15" containerName="istio-proxy" Nov 03 21:10:56.536131 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.536122 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0c46b1-d75f-4f7d-b10d-5899d1770e64" containerName="maas-api" Nov 03 21:10:56.540525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.540505 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.607185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.603693 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk"] Nov 03 21:10:56.617647 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.617617 4227 patch_prober.go:29] interesting pod/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:10:56.618275 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.618230 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 21:10:56.628532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.628503 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-tmp-dir\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.628798 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.628776 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-model-cache\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.628938 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.628921 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-home\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.629072 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.629054 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxhw\" (UniqueName: \"kubernetes.io/projected/ed0805da-b29e-447a-9e8a-59628d2a31d2-kube-api-access-7nxhw\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.629222 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.629204 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0805da-b29e-447a-9e8a-59628d2a31d2-tls-certs\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.629380 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.629362 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-dshm\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.633338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.633304 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c14420\"/\"kube-root-ca.crt\"" Nov 03 21:10:56.633769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.633751 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c14420\"/\"openshift-service-ca.crt\"" Nov 03 21:10:56.640871 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.640059 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c14420\"/\"test-model-c14420-kserve-self-signed-certs\"" Nov 03 21:10:56.640871 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.640714 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c14420\"/\"default-dockercfg-mdc6z\"" Nov 03 21:10:56.809246 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.799462 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-model-cache\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.809519 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.809494 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-home\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.810021 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.809999 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxhw\" (UniqueName: \"kubernetes.io/projected/ed0805da-b29e-447a-9e8a-59628d2a31d2-kube-api-access-7nxhw\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.814508 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.814487 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0805da-b29e-447a-9e8a-59628d2a31d2-tls-certs\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.815095 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.815076 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-dshm\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.818761 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.818737 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-tmp-dir\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.823377 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.800234 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-model-cache\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.823558 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.819144 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-tmp-dir\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.823704 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.809934 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-home\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.835008 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.834987 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0805da-b29e-447a-9e8a-59628d2a31d2-tls-certs\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.850571 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.850547 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-dshm\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:56.921861 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:56.921826 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxhw\" (UniqueName: \"kubernetes.io/projected/ed0805da-b29e-447a-9e8a-59628d2a31d2-kube-api-access-7nxhw\") pod \"test-model-c14420-kserve-69c86d4979-282qk\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:57.199185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:57.198463 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:10:57.615106 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:57.614606 4227 patch_prober.go:29] interesting pod/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:10:57.615106 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:57.614666 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 21:10:57.643523 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:57.643477 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk"] Nov 03 21:10:58.444178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:58.444091 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" event={"ID":"ed0805da-b29e-447a-9e8a-59628d2a31d2","Type":"ContainerStarted","Data":"afe78ac4b871aaca985a6be3687e77938099e53bd2b350a6c0482a9b196cce6d"} Nov 03 21:10:58.655191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:58.642280 4227 patch_prober.go:29] interesting pod/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 21:10:58.655191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:58.642368 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 21:10:58.665186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:58.661947 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28"] Nov 03 21:10:58.699069 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:58.698986 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" podUID="00237c14-2238-4ed7-b408-3984cc0dfece" containerName="maas-api" containerID="cri-o://8cb6310ec58a4d3aa7387d3131fdffd6d26a74d48e4ef0c5653450780a6f9cea" gracePeriod=30 Nov 03 21:10:58.793181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:58.788264 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 21:10:58.878185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:58.864068 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:10:59.170413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.169656 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 03 21:10:59.563552 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.563499 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" event={"ID":"ed0805da-b29e-447a-9e8a-59628d2a31d2","Type":"ContainerStarted","Data":"b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19"} Nov 03 21:10:59.659338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.658984 4227 generic.go:356] "Generic (PLEG): container finished" podID="00237c14-2238-4ed7-b408-3984cc0dfece" containerID="8cb6310ec58a4d3aa7387d3131fdffd6d26a74d48e4ef0c5653450780a6f9cea" exitCode=0 Nov 03 21:10:59.659338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.659113 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" event={"ID":"00237c14-2238-4ed7-b408-3984cc0dfece","Type":"ContainerDied","Data":"8cb6310ec58a4d3aa7387d3131fdffd6d26a74d48e4ef0c5653450780a6f9cea"} Nov 03 21:10:59.733609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.733334 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 21:10:59.894859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.893735 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls\") pod \"00237c14-2238-4ed7-b408-3984cc0dfece\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " Nov 03 21:10:59.894859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.893796 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00237c14-2238-4ed7-b408-3984cc0dfece-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00237c14-2238-4ed7-b408-3984cc0dfece-openshift-service-ca\") pod \"00237c14-2238-4ed7-b408-3984cc0dfece\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " Nov 03 21:10:59.894859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.893911 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00237c14-2238-4ed7-b408-3984cc0dfece-kube-api-access-4fg4h\" (UniqueName: \"kubernetes.io/projected/00237c14-2238-4ed7-b408-3984cc0dfece-kube-api-access-4fg4h\") pod \"00237c14-2238-4ed7-b408-3984cc0dfece\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " Nov 03 21:10:59.894859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.893962 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-metrics-certs\") pod \"00237c14-2238-4ed7-b408-3984cc0dfece\" (UID: \"00237c14-2238-4ed7-b408-3984cc0dfece\") " Nov 03 21:10:59.916927 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.916884 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00237c14-2238-4ed7-b408-3984cc0dfece-openshift-service-ca" pod "00237c14-2238-4ed7-b408-3984cc0dfece" (UID: "00237c14-2238-4ed7-b408-3984cc0dfece"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:59.957008 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.956950 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" podStartSLOduration=3.956928709 podStartE2EDuration="3.956928709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:59.710107628 +0000 UTC m=+2526.947085503" watchObservedRunningTime="2026-11-03 21:10:59.956928709 +0000 UTC m=+2527.193906585" Nov 03 21:10:59.988596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.988536 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00237c14-2238-4ed7-b408-3984cc0dfece-kube-api-access-4fg4h" pod "00237c14-2238-4ed7-b408-3984cc0dfece" (UID: "00237c14-2238-4ed7-b408-3984cc0dfece"). InnerVolumeSpecName "kube-api-access-4fg4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:59.989553 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:10:59.989508 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-metrics-certs" pod "00237c14-2238-4ed7-b408-3984cc0dfece" (UID: "00237c14-2238-4ed7-b408-3984cc0dfece"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:00.019470 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.019068 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fg4h\" (UniqueName: \"kubernetes.io/projected/00237c14-2238-4ed7-b408-3984cc0dfece-kube-api-access-4fg4h\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:00.019470 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.019111 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:00.019470 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.019131 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00237c14-2238-4ed7-b408-3984cc0dfece-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:00.089212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.078250 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g"] Nov 03 21:11:00.089212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.078977 4227 cpu_manager.go:414] "containerMap: removing container" podUID="00237c14-2238-4ed7-b408-3984cc0dfece" containerName="maas-api" Nov 03 21:11:00.089212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.078998 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00237c14-2238-4ed7-b408-3984cc0dfece" containerName="maas-api" Nov 03 21:11:00.089212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.079120 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="00237c14-2238-4ed7-b408-3984cc0dfece" containerName="maas-api" Nov 03 21:11:00.089212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.079140 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fdb9d7-abfd-4c25-9f60-503543c9a0a7" containerName="istio-proxy" Nov 03 21:11:00.107214 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.097420 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:11:00.107214 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.101466 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.119522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.107522 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:11:00.135496 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.135455 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g"] Nov 03 21:11:00.149338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.149312 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls" pod "00237c14-2238-4ed7-b408-3984cc0dfece" (UID: "00237c14-2238-4ed7-b408-3984cc0dfece"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:00.149740 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.149718 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:11:00.214377 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.168133 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-562eb2\"/\"default-dockercfg-4nhvt\"" Nov 03 21:11:00.214377 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.168270 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-562eb2\"/\"kube-root-ca.crt\"" Nov 03 21:11:00.214377 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.168427 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-562eb2\"/\"test-model-562eb2-kserve-self-signed-certs\"" Nov 03 21:11:00.214377 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.210556 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:11:00.215796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.215771 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-562eb2\"/\"openshift-service-ca.crt\"" Nov 03 21:11:00.231596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.225815 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-dshm\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.231596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.225901 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tls-certs\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.231596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.225942 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-home\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.231596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.225970 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxwr\" (UniqueName: \"kubernetes.io/projected/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-kube-api-access-8fxwr\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.231596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.226048 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-model-cache\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.231596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.226075 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tmp-dir\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.231596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.226170 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00237c14-2238-4ed7-b408-3984cc0dfece-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:00.330040 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.329997 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-dshm\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.330389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.330365 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tls-certs\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.330591 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.330566 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-home\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.330732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.330713 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxwr\" (UniqueName: \"kubernetes.io/projected/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-kube-api-access-8fxwr\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.330931 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.330914 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-model-cache\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.331065 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.331049 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tmp-dir\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.331907 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.331886 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tmp-dir\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.332404 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.332376 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-home\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.338759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.338734 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-model-cache\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.359178 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.344263 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tls-certs\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.359371 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.352462 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-dshm\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.401867 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.401792 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxwr\" (UniqueName: \"kubernetes.io/projected/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-kube-api-access-8fxwr\") pod \"test-model-562eb2-kserve-f48ff6c-jkc8g\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.588182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.586343 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:00.744181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.732432 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" Nov 03 21:11:00.752554 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.752024 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28" event={"ID":"00237c14-2238-4ed7-b408-3984cc0dfece","Type":"ContainerDied","Data":"3ff1cae18faee4020fda2bd88aec9fb7d934d620540e6b6661f22666593df074"} Nov 03 21:11:00.752554 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.752094 4227 scope.go:122] "RemoveContainer" containerID="8cb6310ec58a4d3aa7387d3131fdffd6d26a74d48e4ef0c5653450780a6f9cea" Nov 03 21:11:00.937499 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.937465 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28"] Nov 03 21:11:00.958279 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:00.958205 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fd9afe-77c4d7fc5b-mgn28"] Nov 03 21:11:01.169683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:01.169651 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g"] Nov 03 21:11:01.175601 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:11:01.175575 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf15418ca_ed1c_46f7_94d7_3e0d58375ca7.slice/crio-fb3f0c7749c4356a258ffe0731aeadf1ed0d77db61e0f5cab2a3cd46fb10dc39 WatchSource:0}: Error finding container fb3f0c7749c4356a258ffe0731aeadf1ed0d77db61e0f5cab2a3cd46fb10dc39: Status 404 returned error can't find the container with id fb3f0c7749c4356a258ffe0731aeadf1ed0d77db61e0f5cab2a3cd46fb10dc39 Nov 03 21:11:01.739858 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:01.739814 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" event={"ID":"f15418ca-ed1c-46f7-94d7-3e0d58375ca7","Type":"ContainerStarted","Data":"cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783"} Nov 03 21:11:01.740047 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:01.739861 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" event={"ID":"f15418ca-ed1c-46f7-94d7-3e0d58375ca7","Type":"ContainerStarted","Data":"fb3f0c7749c4356a258ffe0731aeadf1ed0d77db61e0f5cab2a3cd46fb10dc39"} Nov 03 21:11:01.847955 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:01.847904 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" podStartSLOduration=2.847887962 podStartE2EDuration="2.847887962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:01.846559932 +0000 UTC m=+2529.083537807" watchObservedRunningTime="2026-11-03 21:11:01.847887962 +0000 UTC m=+2529.084865837" Nov 03 21:11:02.589260 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:02.589221 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:02.900048 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:02.900015 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00237c14-2238-4ed7-b408-3984cc0dfece" path="/var/lib/kubelet/pods/00237c14-2238-4ed7-b408-3984cc0dfece/volumes" Nov 03 21:11:07.199395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:07.199362 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:11:07.211675 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:07.211649 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:11:07.789757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:07.789726 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:11:10.601182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:10.593548 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:10.632434 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:10.632402 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:10.802831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:10.802798 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:11:26.082969 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:11:26.082933 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:11:47.647904 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.647869 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd"] Nov 03 21:11:47.648425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.648074 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" podUID="f29299fe-e752-4a30-bcad-a827cac73147" containerName="maas-api" containerID="cri-o://7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943" gracePeriod=30 Nov 03 21:11:47.995894 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.995866 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 21:11:47.998564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.998534 4227 generic.go:356] "Generic (PLEG): container finished" podID="f29299fe-e752-4a30-bcad-a827cac73147" containerID="7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943" exitCode=0 Nov 03 21:11:47.998691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.998575 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" Nov 03 21:11:47.998691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.998608 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" event={"ID":"f29299fe-e752-4a30-bcad-a827cac73147","Type":"ContainerDied","Data":"7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943"} Nov 03 21:11:47.998691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.998663 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd" event={"ID":"f29299fe-e752-4a30-bcad-a827cac73147","Type":"ContainerDied","Data":"b681c2982652efbbee9d3b70cc0de9285ce570c9105672797ed520c44b8b44e3"} Nov 03 21:11:47.998876 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:47.998692 4227 scope.go:122] "RemoveContainer" containerID="7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943" Nov 03 21:11:48.009430 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.009408 4227 scope.go:122] "RemoveContainer" containerID="7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943" Nov 03 21:11:48.009702 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:11:48.009679 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943\": container with ID starting with 7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943 not found: ID does not exist" containerID="7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943" Nov 03 21:11:48.009780 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.009711 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943"} err="failed to get container status \"7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943\": rpc error: code = NotFound desc = could not find container \"7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943\": container with ID starting with 7e6957cbe5a840ffd543644609f90f836be3747c3ec3c54a945b0ea9a0d58943 not found: ID does not exist" Nov 03 21:11:48.087391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.086498 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29299fe-e752-4a30-bcad-a827cac73147-kube-api-access-gxv2c\" (UniqueName: \"kubernetes.io/projected/f29299fe-e752-4a30-bcad-a827cac73147-kube-api-access-gxv2c\") pod \"f29299fe-e752-4a30-bcad-a827cac73147\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " Nov 03 21:11:48.087391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.086589 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-metrics-certs\") pod \"f29299fe-e752-4a30-bcad-a827cac73147\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " Nov 03 21:11:48.087391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.086723 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls\") pod \"f29299fe-e752-4a30-bcad-a827cac73147\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " Nov 03 21:11:48.087391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.086809 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29299fe-e752-4a30-bcad-a827cac73147-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29299fe-e752-4a30-bcad-a827cac73147-openshift-service-ca\") pod \"f29299fe-e752-4a30-bcad-a827cac73147\" (UID: \"f29299fe-e752-4a30-bcad-a827cac73147\") " Nov 03 21:11:48.088191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.087866 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29299fe-e752-4a30-bcad-a827cac73147-openshift-service-ca" pod "f29299fe-e752-4a30-bcad-a827cac73147" (UID: "f29299fe-e752-4a30-bcad-a827cac73147"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:48.107384 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.105634 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29299fe-e752-4a30-bcad-a827cac73147-kube-api-access-gxv2c" pod "f29299fe-e752-4a30-bcad-a827cac73147" (UID: "f29299fe-e752-4a30-bcad-a827cac73147"). InnerVolumeSpecName "kube-api-access-gxv2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:48.107384 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.105734 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-metrics-certs" pod "f29299fe-e752-4a30-bcad-a827cac73147" (UID: "f29299fe-e752-4a30-bcad-a827cac73147"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:48.107634 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.107514 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls" pod "f29299fe-e752-4a30-bcad-a827cac73147" (UID: "f29299fe-e752-4a30-bcad-a827cac73147"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:48.188501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.188466 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29299fe-e752-4a30-bcad-a827cac73147-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:48.188501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.188493 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxv2c\" (UniqueName: \"kubernetes.io/projected/f29299fe-e752-4a30-bcad-a827cac73147-kube-api-access-gxv2c\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:48.188501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.188508 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:48.188799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.188520 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29299fe-e752-4a30-bcad-a827cac73147-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:48.411670 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.411614 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd"] Nov 03 21:11:48.416918 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.416892 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b119f2-5d5584778d-nsgwd"] Nov 03 21:11:48.899345 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:11:48.899313 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29299fe-e752-4a30-bcad-a827cac73147" path="/var/lib/kubelet/pods/f29299fe-e752-4a30-bcad-a827cac73147/volumes" Nov 03 21:11:56.086881 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:11:56.086845 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:12:07.842098 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:07.842066 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5"] Nov 03 21:12:22.862447 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:22.862405 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" containerID="cri-o://8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416" gracePeriod=45 Nov 03 21:12:23.122387 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.122332 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 21:12:23.226566 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.226529 4227 generic.go:356] "Generic (PLEG): container finished" podID="e68e1972-6268-4597-9520-c1a521a3d998" containerID="8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416" exitCode=0 Nov 03 21:12:23.226722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.226575 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" Nov 03 21:12:23.226722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.226611 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" event={"ID":"e68e1972-6268-4597-9520-c1a521a3d998","Type":"ContainerDied","Data":"8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416"} Nov 03 21:12:23.226722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.226654 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5" event={"ID":"e68e1972-6268-4597-9520-c1a521a3d998","Type":"ContainerDied","Data":"f296c78910f445256e50a0fe9b736a3c23134ab4d4c5e7089207174c94c996e7"} Nov 03 21:12:23.226722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.226673 4227 scope.go:122] "RemoveContainer" containerID="8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416" Nov 03 21:12:23.246925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.246397 4227 scope.go:122] "RemoveContainer" containerID="8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416" Nov 03 21:12:23.246925 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:12:23.246871 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416\": container with ID starting with 8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416 not found: ID does not exist" containerID="8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416" Nov 03 21:12:23.247119 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.246934 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416"} err="failed to get container status \"8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416\": rpc error: code = NotFound desc = could not find container \"8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416\": container with ID starting with 8d6ea660aacac0a427df3e6ced1921c723706337d1ef9b3d09f6f183c9bf0416 not found: ID does not exist" Nov 03 21:12:23.285928 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.285888 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-model-cache\") pod \"e68e1972-6268-4597-9520-c1a521a3d998\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " Nov 03 21:12:23.286103 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.285973 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-dshm\") pod \"e68e1972-6268-4597-9520-c1a521a3d998\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " Nov 03 21:12:23.286103 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.285997 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68e1972-6268-4597-9520-c1a521a3d998-tls-certs\" (UniqueName: \"kubernetes.io/secret/e68e1972-6268-4597-9520-c1a521a3d998-tls-certs\") pod \"e68e1972-6268-4597-9520-c1a521a3d998\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " Nov 03 21:12:23.286103 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286023 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68e1972-6268-4597-9520-c1a521a3d998-kube-api-access-cbpvh\" (UniqueName: \"kubernetes.io/projected/e68e1972-6268-4597-9520-c1a521a3d998-kube-api-access-cbpvh\") pod \"e68e1972-6268-4597-9520-c1a521a3d998\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " Nov 03 21:12:23.286304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286145 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-model-cache" pod "e68e1972-6268-4597-9520-c1a521a3d998" (UID: "e68e1972-6268-4597-9520-c1a521a3d998"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:23.286304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286229 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-tmp-dir\") pod \"e68e1972-6268-4597-9520-c1a521a3d998\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " Nov 03 21:12:23.286413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286302 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-home\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-home\") pod \"e68e1972-6268-4597-9520-c1a521a3d998\" (UID: \"e68e1972-6268-4597-9520-c1a521a3d998\") " Nov 03 21:12:23.286597 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286551 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-tmp-dir" pod "e68e1972-6268-4597-9520-c1a521a3d998" (UID: "e68e1972-6268-4597-9520-c1a521a3d998"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:23.286739 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286721 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-home" pod "e68e1972-6268-4597-9520-c1a521a3d998" (UID: "e68e1972-6268-4597-9520-c1a521a3d998"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:23.286867 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286851 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:23.286925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286872 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:23.286925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.286885 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:23.288543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.288513 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68e1972-6268-4597-9520-c1a521a3d998-tls-certs" pod "e68e1972-6268-4597-9520-c1a521a3d998" (UID: "e68e1972-6268-4597-9520-c1a521a3d998"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:23.288707 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.288686 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68e1972-6268-4597-9520-c1a521a3d998-kube-api-access-cbpvh" pod "e68e1972-6268-4597-9520-c1a521a3d998" (UID: "e68e1972-6268-4597-9520-c1a521a3d998"). InnerVolumeSpecName "kube-api-access-cbpvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:23.288794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.288756 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-dshm" pod "e68e1972-6268-4597-9520-c1a521a3d998" (UID: "e68e1972-6268-4597-9520-c1a521a3d998"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:23.387853 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.387772 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68e1972-6268-4597-9520-c1a521a3d998-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:23.387853 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.387810 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68e1972-6268-4597-9520-c1a521a3d998-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:23.387853 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.387826 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbpvh\" (UniqueName: \"kubernetes.io/projected/e68e1972-6268-4597-9520-c1a521a3d998-kube-api-access-cbpvh\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:23.662005 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.661970 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5"] Nov 03 21:12:23.662228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:23.662028 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/premium-w1-41aa21-kserve-9678c9d89-5b9b5"] Nov 03 21:12:24.899901 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:24.899869 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68e1972-6268-4597-9520-c1a521a3d998" path="/var/lib/kubelet/pods/e68e1972-6268-4597-9520-c1a521a3d998/volumes" Nov 03 21:12:26.081959 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:12:26.081924 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:12:28.248289 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:28.248247 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-lpszt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:12:28.248928 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:12:28.248893 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:12:56.081922 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:12:56.081878 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:13:26.081748 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:13:26.081711 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:13:53.218481 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:13:53.218450 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:13:53.222977 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:13:53.222952 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:13:56.083784 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:13:56.083750 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:14:26.080768 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:14:26.080732 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:14:34.370444 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:34.370357 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk"] Nov 03 21:14:49.392816 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.392774 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" podUID="ed0805da-b29e-447a-9e8a-59628d2a31d2" containerName="main" containerID="cri-o://b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19" gracePeriod=45 Nov 03 21:14:49.652239 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.652217 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:14:49.722474 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722447 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0805da-b29e-447a-9e8a-59628d2a31d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0805da-b29e-447a-9e8a-59628d2a31d2-tls-certs\") pod \"ed0805da-b29e-447a-9e8a-59628d2a31d2\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " Nov 03 21:14:49.722650 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722483 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-dshm\") pod \"ed0805da-b29e-447a-9e8a-59628d2a31d2\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " Nov 03 21:14:49.722650 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722513 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-model-cache\") pod \"ed0805da-b29e-447a-9e8a-59628d2a31d2\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " Nov 03 21:14:49.722650 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722543 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0805da-b29e-447a-9e8a-59628d2a31d2-kube-api-access-7nxhw\" (UniqueName: \"kubernetes.io/projected/ed0805da-b29e-447a-9e8a-59628d2a31d2-kube-api-access-7nxhw\") pod \"ed0805da-b29e-447a-9e8a-59628d2a31d2\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " Nov 03 21:14:49.722650 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722601 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-tmp-dir\") pod \"ed0805da-b29e-447a-9e8a-59628d2a31d2\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " Nov 03 21:14:49.722882 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722721 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-home\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-home\") pod \"ed0805da-b29e-447a-9e8a-59628d2a31d2\" (UID: \"ed0805da-b29e-447a-9e8a-59628d2a31d2\") " Nov 03 21:14:49.722882 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722805 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-model-cache" pod "ed0805da-b29e-447a-9e8a-59628d2a31d2" (UID: "ed0805da-b29e-447a-9e8a-59628d2a31d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:49.722970 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722893 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-tmp-dir" pod "ed0805da-b29e-447a-9e8a-59628d2a31d2" (UID: "ed0805da-b29e-447a-9e8a-59628d2a31d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:49.723015 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.722960 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-home" pod "ed0805da-b29e-447a-9e8a-59628d2a31d2" (UID: "ed0805da-b29e-447a-9e8a-59628d2a31d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:49.723181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.723128 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:49.723181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.723153 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:49.723288 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.723193 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:49.724700 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.724678 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-dshm" pod "ed0805da-b29e-447a-9e8a-59628d2a31d2" (UID: "ed0805da-b29e-447a-9e8a-59628d2a31d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:49.725100 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.725051 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0805da-b29e-447a-9e8a-59628d2a31d2-kube-api-access-7nxhw" pod "ed0805da-b29e-447a-9e8a-59628d2a31d2" (UID: "ed0805da-b29e-447a-9e8a-59628d2a31d2"). InnerVolumeSpecName "kube-api-access-7nxhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:49.725100 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.725092 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0805da-b29e-447a-9e8a-59628d2a31d2-tls-certs" pod "ed0805da-b29e-447a-9e8a-59628d2a31d2" (UID: "ed0805da-b29e-447a-9e8a-59628d2a31d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:49.824235 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.824199 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0805da-b29e-447a-9e8a-59628d2a31d2-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:49.824235 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.824230 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0805da-b29e-447a-9e8a-59628d2a31d2-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:49.824496 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:49.824246 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nxhw\" (UniqueName: \"kubernetes.io/projected/ed0805da-b29e-447a-9e8a-59628d2a31d2-kube-api-access-7nxhw\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:50.093696 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.093612 4227 generic.go:356] "Generic (PLEG): container finished" podID="ed0805da-b29e-447a-9e8a-59628d2a31d2" containerID="b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19" exitCode=0 Nov 03 21:14:50.093696 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.093659 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" Nov 03 21:14:50.093873 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.093694 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" event={"ID":"ed0805da-b29e-447a-9e8a-59628d2a31d2","Type":"ContainerDied","Data":"b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19"} Nov 03 21:14:50.093873 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.093744 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk" event={"ID":"ed0805da-b29e-447a-9e8a-59628d2a31d2","Type":"ContainerDied","Data":"afe78ac4b871aaca985a6be3687e77938099e53bd2b350a6c0482a9b196cce6d"} Nov 03 21:14:50.093873 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.093765 4227 scope.go:122] "RemoveContainer" containerID="b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19" Nov 03 21:14:50.103299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.103277 4227 scope.go:122] "RemoveContainer" containerID="b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19" Nov 03 21:14:50.103537 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:14:50.103515 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19\": container with ID starting with b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19 not found: ID does not exist" containerID="b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19" Nov 03 21:14:50.103590 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.103549 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19"} err="failed to get container status \"b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19\": rpc error: code = NotFound desc = could not find container \"b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19\": container with ID starting with b7d943e921cee37d3243bdf4b1639b02bc5bf92df78a022584edc1a3ce539a19 not found: ID does not exist" Nov 03 21:14:50.245895 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.245865 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk"] Nov 03 21:14:50.292051 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.292024 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c14420/test-model-c14420-kserve-69c86d4979-282qk"] Nov 03 21:14:50.903284 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:14:50.903250 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0805da-b29e-447a-9e8a-59628d2a31d2" path="/var/lib/kubelet/pods/ed0805da-b29e-447a-9e8a-59628d2a31d2/volumes" Nov 03 21:14:56.084029 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:14:56.083986 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:15:00.194190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194148 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb"] Nov 03 21:15:00.194683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194598 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ed0805da-b29e-447a-9e8a-59628d2a31d2" containerName="main" Nov 03 21:15:00.194683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194609 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0805da-b29e-447a-9e8a-59628d2a31d2" containerName="main" Nov 03 21:15:00.194683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194628 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f29299fe-e752-4a30-bcad-a827cac73147" containerName="maas-api" Nov 03 21:15:00.194683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194634 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29299fe-e752-4a30-bcad-a827cac73147" containerName="maas-api" Nov 03 21:15:00.194683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194657 4227 cpu_manager.go:414] "containerMap: removing container" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" Nov 03 21:15:00.194683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194662 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" Nov 03 21:15:00.194900 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194719 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68e1972-6268-4597-9520-c1a521a3d998" containerName="main" Nov 03 21:15:00.194900 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194730 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0805da-b29e-447a-9e8a-59628d2a31d2" containerName="main" Nov 03 21:15:00.194900 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.194737 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29299fe-e752-4a30-bcad-a827cac73147" containerName="maas-api" Nov 03 21:15:00.198712 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.198692 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.225383 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.225361 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r"] Nov 03 21:15:00.228660 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.228645 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.234687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.234635 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv"] Nov 03 21:15:00.238049 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.238032 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.253650 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.253630 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ldx\" (UniqueName: \"kubernetes.io/projected/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-kube-api-access-v6ldx\") pod \"maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.253838 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.253817 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.269728 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.269705 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:15:00.301987 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.301928 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb"] Nov 03 21:15:00.325652 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.325623 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r"] Nov 03 21:15:00.328469 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.328445 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv"] Nov 03 21:15:00.354976 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.354949 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wk8h\" (UniqueName: \"kubernetes.io/projected/53b67219-a600-4166-846f-8b89475bfd36-kube-api-access-7wk8h\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.355095 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.355010 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.355095 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.355076 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d99b9-2716-4795-9c7a-f71259e86d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.355202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.355103 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b67219-a600-4166-846f-8b89475bfd36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.355202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.355126 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kgx\" (UniqueName: \"kubernetes.io/projected/385d99b9-2716-4795-9c7a-f71259e86d6f-kube-api-access-v8kgx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.355202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.355170 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ldx\" (UniqueName: \"kubernetes.io/projected/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-kube-api-access-v6ldx\") pod \"maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.355838 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.355818 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.408464 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.408432 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ldx\" (UniqueName: \"kubernetes.io/projected/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-kube-api-access-v6ldx\") pod \"maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.456115 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.456057 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wk8h\" (UniqueName: \"kubernetes.io/projected/53b67219-a600-4166-846f-8b89475bfd36-kube-api-access-7wk8h\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.456115 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.456103 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d99b9-2716-4795-9c7a-f71259e86d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.456299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.456124 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b67219-a600-4166-846f-8b89475bfd36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.456299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.456153 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kgx\" (UniqueName: \"kubernetes.io/projected/385d99b9-2716-4795-9c7a-f71259e86d6f-kube-api-access-v8kgx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.456782 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.456764 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b67219-a600-4166-846f-8b89475bfd36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.456822 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.456777 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d99b9-2716-4795-9c7a-f71259e86d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.509511 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.509484 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wk8h\" (UniqueName: \"kubernetes.io/projected/53b67219-a600-4166-846f-8b89475bfd36-kube-api-access-7wk8h\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.509648 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.509629 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kgx\" (UniqueName: \"kubernetes.io/projected/385d99b9-2716-4795-9c7a-f71259e86d6f-kube-api-access-v8kgx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.511439 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.511419 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:00.541004 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.540983 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:00.549116 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.549096 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:00.890402 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.890328 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb"] Nov 03 21:15:00.905278 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:15:00.905240 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d154031_55f8_4ac9_93d4_8ea3d7d3a958.slice/crio-9cd6dfd1eab25e24ce96dfef71e26fbd08ae05b9d11085bba0bf6698ab6991cf WatchSource:0}: Error finding container 9cd6dfd1eab25e24ce96dfef71e26fbd08ae05b9d11085bba0bf6698ab6991cf: Status 404 returned error can't find the container with id 9cd6dfd1eab25e24ce96dfef71e26fbd08ae05b9d11085bba0bf6698ab6991cf Nov 03 21:15:00.939750 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.939723 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r"] Nov 03 21:15:00.945990 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:15:00.945965 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385d99b9_2716_4795_9c7a_f71259e86d6f.slice/crio-7fc7bd3c93eb3f04b15edea008f7ec6091ff59e24a09a81244a0dafa609345f8 WatchSource:0}: Error finding container 7fc7bd3c93eb3f04b15edea008f7ec6091ff59e24a09a81244a0dafa609345f8: Status 404 returned error can't find the container with id 7fc7bd3c93eb3f04b15edea008f7ec6091ff59e24a09a81244a0dafa609345f8 Nov 03 21:15:00.950124 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:00.950101 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv"] Nov 03 21:15:00.959682 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:15:00.959656 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b67219_a600_4166_846f_8b89475bfd36.slice/crio-c257c9107d7a6d3f7b0227c5bcfeaa6f79a31b886830c1b00f709ee22cc061ca WatchSource:0}: Error finding container c257c9107d7a6d3f7b0227c5bcfeaa6f79a31b886830c1b00f709ee22cc061ca: Status 404 returned error can't find the container with id c257c9107d7a6d3f7b0227c5bcfeaa6f79a31b886830c1b00f709ee22cc061ca Nov 03 21:15:01.152357 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:01.152308 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" event={"ID":"3d154031-55f8-4ac9-93d4-8ea3d7d3a958","Type":"ContainerStarted","Data":"9cd6dfd1eab25e24ce96dfef71e26fbd08ae05b9d11085bba0bf6698ab6991cf"} Nov 03 21:15:01.153786 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:01.153757 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" event={"ID":"53b67219-a600-4166-846f-8b89475bfd36","Type":"ContainerStarted","Data":"c257c9107d7a6d3f7b0227c5bcfeaa6f79a31b886830c1b00f709ee22cc061ca"} Nov 03 21:15:01.155006 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:01.154984 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" event={"ID":"385d99b9-2716-4795-9c7a-f71259e86d6f","Type":"ContainerStarted","Data":"7fc7bd3c93eb3f04b15edea008f7ec6091ff59e24a09a81244a0dafa609345f8"} Nov 03 21:15:02.165706 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.165665 4227 generic.go:356] "Generic (PLEG): container finished" podID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerID="9588d2e492c26011e9f4b47c52499e27ecbf8e5c2259df661c71cf80ba5a151d" exitCode=60 Nov 03 21:15:02.166204 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.165700 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" event={"ID":"3d154031-55f8-4ac9-93d4-8ea3d7d3a958","Type":"ContainerDied","Data":"9588d2e492c26011e9f4b47c52499e27ecbf8e5c2259df661c71cf80ba5a151d"} Nov 03 21:15:02.166204 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.166062 4227 scope.go:122] "RemoveContainer" containerID="9588d2e492c26011e9f4b47c52499e27ecbf8e5c2259df661c71cf80ba5a151d" Nov 03 21:15:02.167714 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.167622 4227 generic.go:356] "Generic (PLEG): container finished" podID="53b67219-a600-4166-846f-8b89475bfd36" containerID="9fcb2c250edf71b81a82949068b87005d1082788231c273a16a941e921d969d0" exitCode=60 Nov 03 21:15:02.167714 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.167653 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" event={"ID":"53b67219-a600-4166-846f-8b89475bfd36","Type":"ContainerDied","Data":"9fcb2c250edf71b81a82949068b87005d1082788231c273a16a941e921d969d0"} Nov 03 21:15:02.167928 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.167910 4227 scope.go:122] "RemoveContainer" containerID="9fcb2c250edf71b81a82949068b87005d1082788231c273a16a941e921d969d0" Nov 03 21:15:02.169421 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.169352 4227 generic.go:356] "Generic (PLEG): container finished" podID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerID="22eab13acd0fd010a2aef6b706fb2c8c4d007b3b8d8cf14c751f0588d84ad6d1" exitCode=60 Nov 03 21:15:02.169482 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.169427 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" event={"ID":"385d99b9-2716-4795-9c7a-f71259e86d6f","Type":"ContainerDied","Data":"22eab13acd0fd010a2aef6b706fb2c8c4d007b3b8d8cf14c751f0588d84ad6d1"} Nov 03 21:15:02.169688 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:02.169671 4227 scope.go:122] "RemoveContainer" containerID="22eab13acd0fd010a2aef6b706fb2c8c4d007b3b8d8cf14c751f0588d84ad6d1" Nov 03 21:15:03.177569 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.177537 4227 generic.go:356] "Generic (PLEG): container finished" podID="53b67219-a600-4166-846f-8b89475bfd36" containerID="fc08dd12ace5e5d09388bfb6e363e85b5ea3ccde803e75ea24abed16f692c6f3" exitCode=60 Nov 03 21:15:03.178027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.177626 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" event={"ID":"53b67219-a600-4166-846f-8b89475bfd36","Type":"ContainerDied","Data":"fc08dd12ace5e5d09388bfb6e363e85b5ea3ccde803e75ea24abed16f692c6f3"} Nov 03 21:15:03.178027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.177704 4227 scope.go:122] "RemoveContainer" containerID="9fcb2c250edf71b81a82949068b87005d1082788231c273a16a941e921d969d0" Nov 03 21:15:03.178027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.177964 4227 scope.go:122] "RemoveContainer" containerID="fc08dd12ace5e5d09388bfb6e363e85b5ea3ccde803e75ea24abed16f692c6f3" Nov 03 21:15:03.178342 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:03.178316 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv_odh-ai-gateway-infra(53b67219-a600-4166-846f-8b89475bfd36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" podUID="53b67219-a600-4166-846f-8b89475bfd36" Nov 03 21:15:03.180438 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.180405 4227 generic.go:356] "Generic (PLEG): container finished" podID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerID="04844b4e00e2bc4052b21f66e0b9ff3a61d4ea8e656aa60809c4dd6ea536a399" exitCode=60 Nov 03 21:15:03.185936 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.180921 4227 scope.go:122] "RemoveContainer" containerID="04844b4e00e2bc4052b21f66e0b9ff3a61d4ea8e656aa60809c4dd6ea536a399" Nov 03 21:15:03.185936 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:03.181246 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r_odh-ai-gateway-infra(385d99b9-2716-4795-9c7a-f71259e86d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" Nov 03 21:15:03.185936 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.181291 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" event={"ID":"385d99b9-2716-4795-9c7a-f71259e86d6f","Type":"ContainerDied","Data":"04844b4e00e2bc4052b21f66e0b9ff3a61d4ea8e656aa60809c4dd6ea536a399"} Nov 03 21:15:03.188803 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.188779 4227 generic.go:356] "Generic (PLEG): container finished" podID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerID="6f893fc8042fd757c70155f8bbaab03adfadb3be1ed94e23eccb593ec4c36cd5" exitCode=60 Nov 03 21:15:03.188917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.188809 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" event={"ID":"3d154031-55f8-4ac9-93d4-8ea3d7d3a958","Type":"ContainerDied","Data":"6f893fc8042fd757c70155f8bbaab03adfadb3be1ed94e23eccb593ec4c36cd5"} Nov 03 21:15:03.189081 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.189066 4227 scope.go:122] "RemoveContainer" containerID="6f893fc8042fd757c70155f8bbaab03adfadb3be1ed94e23eccb593ec4c36cd5" Nov 03 21:15:03.189298 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:03.189280 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb_odh-ai-gateway-infra(3d154031-55f8-4ac9-93d4-8ea3d7d3a958)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" Nov 03 21:15:03.195371 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.195353 4227 scope.go:122] "RemoveContainer" containerID="22eab13acd0fd010a2aef6b706fb2c8c4d007b3b8d8cf14c751f0588d84ad6d1" Nov 03 21:15:03.206789 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:03.206771 4227 scope.go:122] "RemoveContainer" containerID="9588d2e492c26011e9f4b47c52499e27ecbf8e5c2259df661c71cf80ba5a151d" Nov 03 21:15:04.195525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:04.195499 4227 scope.go:122] "RemoveContainer" containerID="fc08dd12ace5e5d09388bfb6e363e85b5ea3ccde803e75ea24abed16f692c6f3" Nov 03 21:15:04.196007 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:04.195828 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv_odh-ai-gateway-infra(53b67219-a600-4166-846f-8b89475bfd36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" podUID="53b67219-a600-4166-846f-8b89475bfd36" Nov 03 21:15:04.196971 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:04.196950 4227 scope.go:122] "RemoveContainer" containerID="04844b4e00e2bc4052b21f66e0b9ff3a61d4ea8e656aa60809c4dd6ea536a399" Nov 03 21:15:04.197240 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:04.197220 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r_odh-ai-gateway-infra(385d99b9-2716-4795-9c7a-f71259e86d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" Nov 03 21:15:04.198414 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:04.198397 4227 scope.go:122] "RemoveContainer" containerID="6f893fc8042fd757c70155f8bbaab03adfadb3be1ed94e23eccb593ec4c36cd5" Nov 03 21:15:04.198590 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:04.198566 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb_odh-ai-gateway-infra(3d154031-55f8-4ac9-93d4-8ea3d7d3a958)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" Nov 03 21:15:10.613832 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:10.613797 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g"] Nov 03 21:15:14.893225 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:14.893191 4227 scope.go:122] "RemoveContainer" containerID="04844b4e00e2bc4052b21f66e0b9ff3a61d4ea8e656aa60809c4dd6ea536a399" Nov 03 21:15:15.258656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:15.258622 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" event={"ID":"385d99b9-2716-4795-9c7a-f71259e86d6f","Type":"ContainerStarted","Data":"7bf8d1fc1b20c8d060fd8a9909ddd10fdd5027b8e86deb54d5055997265b7005"} Nov 03 21:15:15.366256 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:15.366198 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" podStartSLOduration=15.366183907 podStartE2EDuration="15.366183907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:15.363456174 +0000 UTC m=+2782.600434046" watchObservedRunningTime="2026-11-03 21:15:15.366183907 +0000 UTC m=+2782.603161781" Nov 03 21:15:16.272237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:16.272195 4227 generic.go:356] "Generic (PLEG): container finished" podID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerID="7bf8d1fc1b20c8d060fd8a9909ddd10fdd5027b8e86deb54d5055997265b7005" exitCode=60 Nov 03 21:15:16.272659 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:16.272255 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" event={"ID":"385d99b9-2716-4795-9c7a-f71259e86d6f","Type":"ContainerDied","Data":"7bf8d1fc1b20c8d060fd8a9909ddd10fdd5027b8e86deb54d5055997265b7005"} Nov 03 21:15:16.272659 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:16.272333 4227 scope.go:122] "RemoveContainer" containerID="04844b4e00e2bc4052b21f66e0b9ff3a61d4ea8e656aa60809c4dd6ea536a399" Nov 03 21:15:16.272786 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:16.272756 4227 scope.go:122] "RemoveContainer" containerID="7bf8d1fc1b20c8d060fd8a9909ddd10fdd5027b8e86deb54d5055997265b7005" Nov 03 21:15:16.273088 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:16.273062 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r_odh-ai-gateway-infra(385d99b9-2716-4795-9c7a-f71259e86d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" Nov 03 21:15:16.373895 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:16.373854 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r"] Nov 03 21:15:16.893190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:16.893144 4227 scope.go:122] "RemoveContainer" containerID="fc08dd12ace5e5d09388bfb6e363e85b5ea3ccde803e75ea24abed16f692c6f3" Nov 03 21:15:17.279438 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.279403 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" event={"ID":"53b67219-a600-4166-846f-8b89475bfd36","Type":"ContainerStarted","Data":"bf0b5113b9575207b44c5f54d9bcb90f1c34a1951fd7f6a180454ec46ac1cf1d"} Nov 03 21:15:17.385858 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.385797 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" podStartSLOduration=17.385777066 podStartE2EDuration="17.385777066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:17.38493301 +0000 UTC m=+2784.621910899" watchObservedRunningTime="2026-11-03 21:15:17.385777066 +0000 UTC m=+2784.622754942" Nov 03 21:15:17.422785 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.422761 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:17.464140 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.464113 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385d99b9-2716-4795-9c7a-f71259e86d6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d99b9-2716-4795-9c7a-f71259e86d6f-maas-api-service-ca\") pod \"385d99b9-2716-4795-9c7a-f71259e86d6f\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " Nov 03 21:15:17.464263 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.464208 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385d99b9-2716-4795-9c7a-f71259e86d6f-kube-api-access-v8kgx\" (UniqueName: \"kubernetes.io/projected/385d99b9-2716-4795-9c7a-f71259e86d6f-kube-api-access-v8kgx\") pod \"385d99b9-2716-4795-9c7a-f71259e86d6f\" (UID: \"385d99b9-2716-4795-9c7a-f71259e86d6f\") " Nov 03 21:15:17.464510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.464484 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385d99b9-2716-4795-9c7a-f71259e86d6f-maas-api-service-ca" pod "385d99b9-2716-4795-9c7a-f71259e86d6f" (UID: "385d99b9-2716-4795-9c7a-f71259e86d6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:17.466107 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.466089 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385d99b9-2716-4795-9c7a-f71259e86d6f-kube-api-access-v8kgx" pod "385d99b9-2716-4795-9c7a-f71259e86d6f" (UID: "385d99b9-2716-4795-9c7a-f71259e86d6f"). InnerVolumeSpecName "kube-api-access-v8kgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:17.565471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.565448 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385d99b9-2716-4795-9c7a-f71259e86d6f-maas-api-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:17.565471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.565471 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kgx\" (UniqueName: \"kubernetes.io/projected/385d99b9-2716-4795-9c7a-f71259e86d6f-kube-api-access-v8kgx\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:17.893087 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:17.893063 4227 scope.go:122] "RemoveContainer" containerID="6f893fc8042fd757c70155f8bbaab03adfadb3be1ed94e23eccb593ec4c36cd5" Nov 03 21:15:18.297242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.297205 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" event={"ID":"3d154031-55f8-4ac9-93d4-8ea3d7d3a958","Type":"ContainerStarted","Data":"eab3466cc0436f0fe512db4e85c8232af9e191496a88a7511787d45faed674c6"} Nov 03 21:15:18.299475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.299440 4227 generic.go:356] "Generic (PLEG): container finished" podID="53b67219-a600-4166-846f-8b89475bfd36" containerID="bf0b5113b9575207b44c5f54d9bcb90f1c34a1951fd7f6a180454ec46ac1cf1d" exitCode=60 Nov 03 21:15:18.299614 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.299534 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" event={"ID":"53b67219-a600-4166-846f-8b89475bfd36","Type":"ContainerDied","Data":"bf0b5113b9575207b44c5f54d9bcb90f1c34a1951fd7f6a180454ec46ac1cf1d"} Nov 03 21:15:18.299686 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.299632 4227 scope.go:122] "RemoveContainer" containerID="fc08dd12ace5e5d09388bfb6e363e85b5ea3ccde803e75ea24abed16f692c6f3" Nov 03 21:15:18.299966 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.299949 4227 scope.go:122] "RemoveContainer" containerID="bf0b5113b9575207b44c5f54d9bcb90f1c34a1951fd7f6a180454ec46ac1cf1d" Nov 03 21:15:18.300219 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:18.300195 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv_odh-ai-gateway-infra(53b67219-a600-4166-846f-8b89475bfd36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" podUID="53b67219-a600-4166-846f-8b89475bfd36" Nov 03 21:15:18.300946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.300926 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" event={"ID":"385d99b9-2716-4795-9c7a-f71259e86d6f","Type":"ContainerDied","Data":"7fc7bd3c93eb3f04b15edea008f7ec6091ff59e24a09a81244a0dafa609345f8"} Nov 03 21:15:18.301005 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.300959 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r" Nov 03 21:15:18.311391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.311370 4227 scope.go:122] "RemoveContainer" containerID="7bf8d1fc1b20c8d060fd8a9909ddd10fdd5027b8e86deb54d5055997265b7005" Nov 03 21:15:18.395691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.395662 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv"] Nov 03 21:15:18.405872 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.405828 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" podStartSLOduration=18.405815031 podStartE2EDuration="18.405815031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:18.403846513 +0000 UTC m=+2785.640824386" watchObservedRunningTime="2026-11-03 21:15:18.405815031 +0000 UTC m=+2785.642792905" Nov 03 21:15:18.509732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.509702 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r"] Nov 03 21:15:18.517171 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.517139 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895675-84h8r"] Nov 03 21:15:18.900612 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:18.900581 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" path="/var/lib/kubelet/pods/385d99b9-2716-4795-9c7a-f71259e86d6f/volumes" Nov 03 21:15:19.311546 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.311517 4227 generic.go:356] "Generic (PLEG): container finished" podID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerID="eab3466cc0436f0fe512db4e85c8232af9e191496a88a7511787d45faed674c6" exitCode=60 Nov 03 21:15:19.311864 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.311585 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" event={"ID":"3d154031-55f8-4ac9-93d4-8ea3d7d3a958","Type":"ContainerDied","Data":"eab3466cc0436f0fe512db4e85c8232af9e191496a88a7511787d45faed674c6"} Nov 03 21:15:19.311864 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.311645 4227 scope.go:122] "RemoveContainer" containerID="6f893fc8042fd757c70155f8bbaab03adfadb3be1ed94e23eccb593ec4c36cd5" Nov 03 21:15:19.312007 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.311994 4227 scope.go:122] "RemoveContainer" containerID="eab3466cc0436f0fe512db4e85c8232af9e191496a88a7511787d45faed674c6" Nov 03 21:15:19.312301 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:19.312282 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb_odh-ai-gateway-infra(3d154031-55f8-4ac9-93d4-8ea3d7d3a958)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" Nov 03 21:15:19.415077 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.415043 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb"] Nov 03 21:15:19.466374 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.466346 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:19.489717 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.489687 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b67219-a600-4166-846f-8b89475bfd36-kube-api-access-7wk8h\" (UniqueName: \"kubernetes.io/projected/53b67219-a600-4166-846f-8b89475bfd36-kube-api-access-7wk8h\") pod \"53b67219-a600-4166-846f-8b89475bfd36\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " Nov 03 21:15:19.489853 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.489792 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b67219-a600-4166-846f-8b89475bfd36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b67219-a600-4166-846f-8b89475bfd36-maas-api-service-ca\") pod \"53b67219-a600-4166-846f-8b89475bfd36\" (UID: \"53b67219-a600-4166-846f-8b89475bfd36\") " Nov 03 21:15:19.490285 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.490084 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b67219-a600-4166-846f-8b89475bfd36-maas-api-service-ca" pod "53b67219-a600-4166-846f-8b89475bfd36" (UID: "53b67219-a600-4166-846f-8b89475bfd36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:19.491824 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.491802 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b67219-a600-4166-846f-8b89475bfd36-kube-api-access-7wk8h" pod "53b67219-a600-4166-846f-8b89475bfd36" (UID: "53b67219-a600-4166-846f-8b89475bfd36"). InnerVolumeSpecName "kube-api-access-7wk8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:19.591979 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.591901 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wk8h\" (UniqueName: \"kubernetes.io/projected/53b67219-a600-4166-846f-8b89475bfd36-kube-api-access-7wk8h\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:19.591979 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:19.591936 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b67219-a600-4166-846f-8b89475bfd36-maas-api-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.319636 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.319609 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" Nov 03 21:15:20.319636 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.319625 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv" event={"ID":"53b67219-a600-4166-846f-8b89475bfd36","Type":"ContainerDied","Data":"c257c9107d7a6d3f7b0227c5bcfeaa6f79a31b886830c1b00f709ee22cc061ca"} Nov 03 21:15:20.320137 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.319684 4227 scope.go:122] "RemoveContainer" containerID="bf0b5113b9575207b44c5f54d9bcb90f1c34a1951fd7f6a180454ec46ac1cf1d" Nov 03 21:15:20.457888 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.457863 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:20.502138 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.502109 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-kube-api-access-v6ldx\" (UniqueName: \"kubernetes.io/projected/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-kube-api-access-v6ldx\") pod \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " Nov 03 21:15:20.502297 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.502242 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-maas-api-service-ca\") pod \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\" (UID: \"3d154031-55f8-4ac9-93d4-8ea3d7d3a958\") " Nov 03 21:15:20.502562 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.502541 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-maas-api-service-ca" pod "3d154031-55f8-4ac9-93d4-8ea3d7d3a958" (UID: "3d154031-55f8-4ac9-93d4-8ea3d7d3a958"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:20.502668 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.502654 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-maas-api-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.504236 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.504214 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-kube-api-access-v6ldx" pod "3d154031-55f8-4ac9-93d4-8ea3d7d3a958" (UID: "3d154031-55f8-4ac9-93d4-8ea3d7d3a958"). InnerVolumeSpecName "kube-api-access-v6ldx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:20.526412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.526389 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv"] Nov 03 21:15:20.532012 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.531990 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895675-6w9pv"] Nov 03 21:15:20.604105 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.604041 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ldx\" (UniqueName: \"kubernetes.io/projected/3d154031-55f8-4ac9-93d4-8ea3d7d3a958-kube-api-access-v6ldx\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.900126 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:20.900100 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b67219-a600-4166-846f-8b89475bfd36" path="/var/lib/kubelet/pods/53b67219-a600-4166-846f-8b89475bfd36/volumes" Nov 03 21:15:21.326899 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:21.326821 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" event={"ID":"3d154031-55f8-4ac9-93d4-8ea3d7d3a958","Type":"ContainerDied","Data":"9cd6dfd1eab25e24ce96dfef71e26fbd08ae05b9d11085bba0bf6698ab6991cf"} Nov 03 21:15:21.326899 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:21.326879 4227 scope.go:122] "RemoveContainer" containerID="eab3466cc0436f0fe512db4e85c8232af9e191496a88a7511787d45faed674c6" Nov 03 21:15:21.327381 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:21.326833 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb" Nov 03 21:15:21.478921 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:21.478892 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb"] Nov 03 21:15:21.524920 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:21.524893 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c14420-29895675-8pqzb"] Nov 03 21:15:22.738813 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:22.738779 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc"] Nov 03 21:15:22.849434 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:22.849399 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk"] Nov 03 21:15:22.899776 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:22.899747 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" path="/var/lib/kubelet/pods/3d154031-55f8-4ac9-93d4-8ea3d7d3a958/volumes" Nov 03 21:15:25.633860 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.633816 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" podUID="f15418ca-ed1c-46f7-94d7-3e0d58375ca7" containerName="main" containerID="cri-o://cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783" gracePeriod=45 Nov 03 21:15:25.792779 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.792743 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" podUID="f15418ca-ed1c-46f7-94d7-3e0d58375ca7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.117:8000/ready\": dial tcp 10.134.0.117:8000: connect: connection refused" Nov 03 21:15:25.900788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.900766 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:15:25.959281 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959255 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tmp-dir\") pod \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " Nov 03 21:15:25.959435 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959316 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-model-cache\") pod \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " Nov 03 21:15:25.959435 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959362 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tls-certs\") pod \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " Nov 03 21:15:25.959435 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959412 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-dshm\") pod \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " Nov 03 21:15:25.959623 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959511 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-kube-api-access-8fxwr\" (UniqueName: \"kubernetes.io/projected/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-kube-api-access-8fxwr\") pod \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " Nov 03 21:15:25.959623 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959542 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tmp-dir" pod "f15418ca-ed1c-46f7-94d7-3e0d58375ca7" (UID: "f15418ca-ed1c-46f7-94d7-3e0d58375ca7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:25.959623 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959549 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-home\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-home\") pod \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\" (UID: \"f15418ca-ed1c-46f7-94d7-3e0d58375ca7\") " Nov 03 21:15:25.959623 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959562 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-model-cache" pod "f15418ca-ed1c-46f7-94d7-3e0d58375ca7" (UID: "f15418ca-ed1c-46f7-94d7-3e0d58375ca7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:25.959842 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.959776 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-home" pod "f15418ca-ed1c-46f7-94d7-3e0d58375ca7" (UID: "f15418ca-ed1c-46f7-94d7-3e0d58375ca7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:25.960147 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.960072 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:25.960147 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.960098 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:25.960147 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.960112 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:25.961796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.961773 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-kube-api-access-8fxwr" pod "f15418ca-ed1c-46f7-94d7-3e0d58375ca7" (UID: "f15418ca-ed1c-46f7-94d7-3e0d58375ca7"). InnerVolumeSpecName "kube-api-access-8fxwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:25.961907 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.961879 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tls-certs" pod "f15418ca-ed1c-46f7-94d7-3e0d58375ca7" (UID: "f15418ca-ed1c-46f7-94d7-3e0d58375ca7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:25.961907 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:25.961876 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-dshm" pod "f15418ca-ed1c-46f7-94d7-3e0d58375ca7" (UID: "f15418ca-ed1c-46f7-94d7-3e0d58375ca7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:26.061701 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.061673 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.061701 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.061699 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxwr\" (UniqueName: \"kubernetes.io/projected/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-kube-api-access-8fxwr\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.061701 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.061710 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15418ca-ed1c-46f7-94d7-3e0d58375ca7-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:26.082568 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:26.082529 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:15:26.364989 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.364913 4227 generic.go:356] "Generic (PLEG): container finished" podID="f15418ca-ed1c-46f7-94d7-3e0d58375ca7" containerID="cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783" exitCode=0 Nov 03 21:15:26.364989 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.364969 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" Nov 03 21:15:26.365204 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.364998 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" event={"ID":"f15418ca-ed1c-46f7-94d7-3e0d58375ca7","Type":"ContainerDied","Data":"cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783"} Nov 03 21:15:26.365204 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.365037 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g" event={"ID":"f15418ca-ed1c-46f7-94d7-3e0d58375ca7","Type":"ContainerDied","Data":"fb3f0c7749c4356a258ffe0731aeadf1ed0d77db61e0f5cab2a3cd46fb10dc39"} Nov 03 21:15:26.365204 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.365056 4227 scope.go:122] "RemoveContainer" containerID="cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783" Nov 03 21:15:26.374056 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.374040 4227 scope.go:122] "RemoveContainer" containerID="cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783" Nov 03 21:15:26.374368 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:26.374348 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783\": container with ID starting with cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783 not found: ID does not exist" containerID="cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783" Nov 03 21:15:26.374441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.374378 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783"} err="failed to get container status \"cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783\": rpc error: code = NotFound desc = could not find container \"cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783\": container with ID starting with cd7c2a2c78bf2cd15dbd1a0c7c1863031a01d42e13a3585979e89a3eabcca783 not found: ID does not exist" Nov 03 21:15:26.518214 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.518185 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g"] Nov 03 21:15:26.564612 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.564578 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-562eb2/test-model-562eb2-kserve-f48ff6c-jkc8g"] Nov 03 21:15:26.899615 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:26.899576 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15418ca-ed1c-46f7-94d7-3e0d58375ca7" path="/var/lib/kubelet/pods/f15418ca-ed1c-46f7-94d7-3e0d58375ca7/volumes" Nov 03 21:15:27.740033 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:27.739995 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" podUID="da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" containerName="payload-processing" containerID="cri-o://7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df" gracePeriod=25 Nov 03 21:15:27.849836 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:27.849800 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" podUID="bd24478d-4d06-4727-a7e9-7b392025bd96" containerName="payload-pre-processing" containerID="cri-o://8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80" gracePeriod=25 Nov 03 21:15:27.987345 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:27.987316 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:15:28.081884 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.081809 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-plugins-config-volume\") pod \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " Nov 03 21:15:28.081884 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.081859 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-kube-api-access-xmx4c\" (UniqueName: \"kubernetes.io/projected/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-kube-api-access-xmx4c\") pod \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\" (UID: \"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c\") " Nov 03 21:15:28.082304 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.082277 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-plugins-config-volume" pod "da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" (UID: "da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:28.084126 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.084098 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-kube-api-access-xmx4c" pod "da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" (UID: "da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c"). InnerVolumeSpecName "kube-api-access-xmx4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:28.183812 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.183779 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.183812 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.183805 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmx4c\" (UniqueName: \"kubernetes.io/projected/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c-kube-api-access-xmx4c\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.377124 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.377035 4227 generic.go:356] "Generic (PLEG): container finished" podID="da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" containerID="7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df" exitCode=0 Nov 03 21:15:28.377124 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.377093 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" Nov 03 21:15:28.377321 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.377130 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" event={"ID":"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c","Type":"ContainerDied","Data":"7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df"} Nov 03 21:15:28.377321 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.377212 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc" event={"ID":"da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c","Type":"ContainerDied","Data":"6785509ce301199577d4df9345b7a8d3c8514f62a318334a06a5563292e2f590"} Nov 03 21:15:28.377321 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.377244 4227 scope.go:122] "RemoveContainer" containerID="7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df" Nov 03 21:15:28.387872 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.387847 4227 scope.go:122] "RemoveContainer" containerID="7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df" Nov 03 21:15:28.388147 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:28.388115 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df\": container with ID starting with 7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df not found: ID does not exist" containerID="7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df" Nov 03 21:15:28.388276 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.388171 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df"} err="failed to get container status \"7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df\": rpc error: code = NotFound desc = could not find container \"7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df\": container with ID starting with 7612ab03c0fd19ef3ea4bbf48ae2be9cce636617609ad6046d1c2df474bfd4df not found: ID does not exist" Nov 03 21:15:28.529307 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.529272 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc"] Nov 03 21:15:28.534376 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.534348 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c14420-7bf59f5b8f-gbsgc"] Nov 03 21:15:28.900539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:28.900507 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" path="/var/lib/kubelet/pods/da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c/volumes" Nov 03 21:15:35.101869 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:35.101836 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c"] Nov 03 21:15:35.102497 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:35.102028 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" containerID="cri-o://4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65" gracePeriod=30 Nov 03 21:15:35.768432 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:35.768387 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" podUID="bd24478d-4d06-4727-a7e9-7b392025bd96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:15:40.351201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.351151 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 21:15:40.417370 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417336 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-credential-socket\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417565 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417387 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-certs\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417565 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417430 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istiod-ca-cert\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417565 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417501 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-data\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417565 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417535 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-envoy\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417565 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417563 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-socket\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417857 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417601 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-token\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-token\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417857 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417652 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-podinfo\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.417857 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417650 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-credential-socket" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:40.417857 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.417682 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-kube-api-access-lgs5p\") pod \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\" (UID: \"93102cbf-09e6-4941-b8e6-d59cc01d8bee\") " Nov 03 21:15:40.418081 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.418049 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.418281 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.418254 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-socket" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:40.418422 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.418374 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-certs" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:40.418422 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.418398 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-data" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:40.418566 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.418452 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istiod-ca-cert" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:40.419910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.419883 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-envoy" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:40.420203 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.420174 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-token" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:40.420318 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.420257 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-podinfo" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:40.420742 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.420724 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-kube-api-access-lgs5p" pod "93102cbf-09e6-4941-b8e6-d59cc01d8bee" (UID: "93102cbf-09e6-4941-b8e6-d59cc01d8bee"). InnerVolumeSpecName "kube-api-access-lgs5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:40.441447 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.441404 4227 generic.go:356] "Generic (PLEG): container finished" podID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerID="4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65" exitCode=0 Nov 03 21:15:40.441578 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.441546 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" event={"ID":"93102cbf-09e6-4941-b8e6-d59cc01d8bee","Type":"ContainerDied","Data":"4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65"} Nov 03 21:15:40.441656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.441634 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" event={"ID":"93102cbf-09e6-4941-b8e6-d59cc01d8bee","Type":"ContainerDied","Data":"62c89f9fdfbd88149c793f218d4788d6382895af53fae8a8e9075f8e157803f2"} Nov 03 21:15:40.441656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.441559 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c" Nov 03 21:15:40.441926 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.441690 4227 scope.go:122] "RemoveContainer" containerID="4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65" Nov 03 21:15:40.456619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.456598 4227 scope.go:122] "RemoveContainer" containerID="4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65" Nov 03 21:15:40.456955 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:40.456932 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65\": container with ID starting with 4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65 not found: ID does not exist" containerID="4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65" Nov 03 21:15:40.457042 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.456970 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65"} err="failed to get container status \"4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65\": rpc error: code = NotFound desc = could not find container \"4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65\": container with ID starting with 4ccb5ef5c40f2dcc41d5fb82bbb3996800caf0031b1eed67ec3c731e8c167e65 not found: ID does not exist" Nov 03 21:15:40.518894 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518865 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.518894 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518891 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-kube-api-access-lgs5p\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.519110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518903 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.519110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518912 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.519110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518921 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.519110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518930 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.519110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518938 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93102cbf-09e6-4941-b8e6-d59cc01d8bee-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.519110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.518947 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93102cbf-09e6-4941-b8e6-d59cc01d8bee-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:40.615250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.615217 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c"] Nov 03 21:15:40.619017 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.618987 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dfl8c"] Nov 03 21:15:40.899340 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:40.899312 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" path="/var/lib/kubelet/pods/93102cbf-09e6-4941-b8e6-d59cc01d8bee/volumes" Nov 03 21:15:45.767970 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:45.767929 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" podUID="bd24478d-4d06-4727-a7e9-7b392025bd96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:15:48.121555 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.121530 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:15:48.238182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.233592 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd24478d-4d06-4727-a7e9-7b392025bd96-kube-api-access-zcb5l\" (UniqueName: \"kubernetes.io/projected/bd24478d-4d06-4727-a7e9-7b392025bd96-kube-api-access-zcb5l\") pod \"bd24478d-4d06-4727-a7e9-7b392025bd96\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " Nov 03 21:15:48.238182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.233710 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd24478d-4d06-4727-a7e9-7b392025bd96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd24478d-4d06-4727-a7e9-7b392025bd96-plugins-config-volume\") pod \"bd24478d-4d06-4727-a7e9-7b392025bd96\" (UID: \"bd24478d-4d06-4727-a7e9-7b392025bd96\") " Nov 03 21:15:48.238182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.234487 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd24478d-4d06-4727-a7e9-7b392025bd96-plugins-config-volume" pod "bd24478d-4d06-4727-a7e9-7b392025bd96" (UID: "bd24478d-4d06-4727-a7e9-7b392025bd96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:48.256476 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.256411 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd24478d-4d06-4727-a7e9-7b392025bd96-kube-api-access-zcb5l" pod "bd24478d-4d06-4727-a7e9-7b392025bd96" (UID: "bd24478d-4d06-4727-a7e9-7b392025bd96"). InnerVolumeSpecName "kube-api-access-zcb5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:48.335593 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.335559 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcb5l\" (UniqueName: \"kubernetes.io/projected/bd24478d-4d06-4727-a7e9-7b392025bd96-kube-api-access-zcb5l\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:48.335593 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.335587 4227 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd24478d-4d06-4727-a7e9-7b392025bd96-plugins-config-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:48.531731 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.531697 4227 generic.go:356] "Generic (PLEG): container finished" podID="bd24478d-4d06-4727-a7e9-7b392025bd96" containerID="8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80" exitCode=0 Nov 03 21:15:48.531910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.531743 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" Nov 03 21:15:48.531910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.531776 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" event={"ID":"bd24478d-4d06-4727-a7e9-7b392025bd96","Type":"ContainerDied","Data":"8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80"} Nov 03 21:15:48.531910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.531825 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk" event={"ID":"bd24478d-4d06-4727-a7e9-7b392025bd96","Type":"ContainerDied","Data":"a414e7a6a486e7c0510bb88aa7fd7fcc4baee8a4be8e082e2637450192150b21"} Nov 03 21:15:48.531910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.531845 4227 scope.go:122] "RemoveContainer" containerID="8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80" Nov 03 21:15:48.540769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.540754 4227 scope.go:122] "RemoveContainer" containerID="8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80" Nov 03 21:15:48.541039 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:48.541017 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80\": container with ID starting with 8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80 not found: ID does not exist" containerID="8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80" Nov 03 21:15:48.541097 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.541048 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80"} err="failed to get container status \"8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80\": rpc error: code = NotFound desc = could not find container \"8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80\": container with ID starting with 8c74033d5ea4d6b9cceace89aea14d8343bba06810f4147d985618469ec80d80 not found: ID does not exist" Nov 03 21:15:48.684816 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.684783 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk"] Nov 03 21:15:48.686769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.686740 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c14420-7c5b58d59-7vxfk"] Nov 03 21:15:48.898724 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:48.898652 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd24478d-4d06-4727-a7e9-7b392025bd96" path="/var/lib/kubelet/pods/bd24478d-4d06-4727-a7e9-7b392025bd96/volumes" Nov 03 21:15:55.867522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:55.867487 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh"] Nov 03 21:15:55.868065 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:55.867745 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" containerID="cri-o://0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b" gracePeriod=30 Nov 03 21:15:56.081539 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:15:56.081505 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:15:59.712003 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:59.711966 4227 patch_prober.go:29] interesting pod/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:15:59.712444 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:15:59.712055 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:16:01.123856 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.123830 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:16:01.307171 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307134 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-kube-api-access-bxms9\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-kube-api-access-bxms9\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.307336 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307297 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-podinfo\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.307393 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307346 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c84ab14-c774-41c4-852a-4eb8698c83ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c84ab14-c774-41c4-852a-4eb8698c83ea-istiod-ca-cert\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.307444 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307397 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-socket\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.307600 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307563 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-credential-socket\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.307754 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307703 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-socket" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:01.307839 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307809 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-credential-socket" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:01.307887 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.307860 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c84ab14-c774-41c4-852a-4eb8698c83ea-istiod-ca-cert" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:01.308039 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308006 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-envoy\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.308191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308076 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-token\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-token\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.308191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308102 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-data\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.308191 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308121 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-certs\") pod \"9c84ab14-c774-41c4-852a-4eb8698c83ea\" (UID: \"9c84ab14-c774-41c4-852a-4eb8698c83ea\") " Nov 03 21:16:01.308606 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308558 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.308759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308728 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.308759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308751 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c84ab14-c774-41c4-852a-4eb8698c83ea-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.308759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308637 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-data" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:01.308976 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.308682 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-certs" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:01.310098 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.310071 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-kube-api-access-bxms9" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "kube-api-access-bxms9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:01.310320 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.310300 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-token" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:01.310320 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.310305 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-podinfo" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:16:01.310450 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.310387 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-envoy" pod "9c84ab14-c774-41c4-852a-4eb8698c83ea" (UID: "9c84ab14-c774-41c4-852a-4eb8698c83ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:01.409320 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.409292 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.409320 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.409316 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.409475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.409329 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.409475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.409342 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxms9\" (UniqueName: \"kubernetes.io/projected/9c84ab14-c774-41c4-852a-4eb8698c83ea-kube-api-access-bxms9\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.409475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.409357 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.409475 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.409371 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c84ab14-c774-41c4-852a-4eb8698c83ea-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:01.609747 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.609663 4227 generic.go:356] "Generic (PLEG): container finished" podID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerID="0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b" exitCode=0 Nov 03 21:16:01.609747 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.609721 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" Nov 03 21:16:01.609933 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.609744 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" event={"ID":"9c84ab14-c774-41c4-852a-4eb8698c83ea","Type":"ContainerDied","Data":"0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b"} Nov 03 21:16:01.609933 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.609822 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh" event={"ID":"9c84ab14-c774-41c4-852a-4eb8698c83ea","Type":"ContainerDied","Data":"6bddd68463084c49485c018f439d05273b6c2c54bc28f5310f8dc0f135c10a4a"} Nov 03 21:16:01.609933 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.609845 4227 scope.go:122] "RemoveContainer" containerID="0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b" Nov 03 21:16:01.619889 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.619870 4227 scope.go:122] "RemoveContainer" containerID="83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975" Nov 03 21:16:01.628557 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.628529 4227 scope.go:122] "RemoveContainer" containerID="0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b" Nov 03 21:16:01.628828 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:01.628809 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b\": container with ID starting with 0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b not found: ID does not exist" containerID="0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b" Nov 03 21:16:01.628893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.628836 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b"} err="failed to get container status \"0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b\": rpc error: code = NotFound desc = could not find container \"0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b\": container with ID starting with 0d089379f50dde8662f3100fc2e8d219db6abff0a0c1b949bd6a6ab29f35546b not found: ID does not exist" Nov 03 21:16:01.628893 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.628854 4227 scope.go:122] "RemoveContainer" containerID="83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975" Nov 03 21:16:01.629093 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:01.629077 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975\": container with ID starting with 83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975 not found: ID does not exist" containerID="83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975" Nov 03 21:16:01.629144 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.629097 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975"} err="failed to get container status \"83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975\": rpc error: code = NotFound desc = could not find container \"83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975\": container with ID starting with 83574f2cffc752fd964002dc069d6fd11de98ac02809a6754bd96fbb8ad86975 not found: ID does not exist" Nov 03 21:16:01.763394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.763357 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh"] Nov 03 21:16:01.810378 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:01.810343 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c34ff-openshift-default-59c6848577-fgjvh"] Nov 03 21:16:02.899798 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:02.899766 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" path="/var/lib/kubelet/pods/9c84ab14-c774-41c4-852a-4eb8698c83ea/volumes" Nov 03 21:16:26.086446 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:26.086398 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:16:26.798499 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:26.798459 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr"] Nov 03 21:16:26.798779 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:26.798749 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" containerID="cri-o://01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c" gracePeriod=30 Nov 03 21:16:28.831454 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:28.831419 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t"] Nov 03 21:16:28.831878 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:28.831654 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" containerID="cri-o://302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0" gracePeriod=30 Nov 03 21:16:30.100854 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:30.100812 4227 patch_prober.go:29] interesting pod/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:16:30.101250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:30.100888 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:16:31.840361 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:31.840278 4227 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca7467_f49f_4151_85fc_5fc490e9a518.slice/crio-01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca7467_f49f_4151_85fc_5fc490e9a518.slice/crio-conmon-01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:16:31.840754 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:31.840612 4227 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca7467_f49f_4151_85fc_5fc490e9a518.slice/crio-conmon-01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca7467_f49f_4151_85fc_5fc490e9a518.slice/crio-01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:16:32.060275 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.060250 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:16:32.097362 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097309 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-data\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097371 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-envoy\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097392 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-token\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-token\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097442 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-podinfo\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097471 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097467 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ca7467-f49f-4151-85fc-5fc490e9a518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ca7467-f49f-4151-85fc-5fc490e9a518-istiod-ca-cert\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097698 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097498 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-credential-socket\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097698 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097540 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-kube-api-access-vfvq7\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-kube-api-access-vfvq7\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097698 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097570 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-socket\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097698 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097587 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-data" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:32.097914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097750 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-certs\") pod \"f6ca7467-f49f-4151-85fc-5fc490e9a518\" (UID: \"f6ca7467-f49f-4151-85fc-5fc490e9a518\") " Nov 03 21:16:32.097914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097880 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-credential-socket" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:32.097914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.097896 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ca7467-f49f-4151-85fc-5fc490e9a518-istiod-ca-cert" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:32.098075 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.098041 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-socket" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:32.098075 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.098068 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-certs" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:32.098298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.098274 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.098298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.098298 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ca7467-f49f-4151-85fc-5fc490e9a518-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.098487 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.098309 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.098487 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.098319 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.098487 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.098327 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.099708 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.099679 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-envoy" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:32.099799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.099761 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-token" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:32.099988 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.099967 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-podinfo" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:16:32.099988 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.099978 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-kube-api-access-vfvq7" pod "f6ca7467-f49f-4151-85fc-5fc490e9a518" (UID: "f6ca7467-f49f-4151-85fc-5fc490e9a518"). InnerVolumeSpecName "kube-api-access-vfvq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:32.199803 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.199766 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.199803 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.199797 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvq7\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-kube-api-access-vfvq7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.199987 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.199813 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.199987 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.199825 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ca7467-f49f-4151-85fc-5fc490e9a518-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:32.784303 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.784269 4227 generic.go:356] "Generic (PLEG): container finished" podID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerID="01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c" exitCode=0 Nov 03 21:16:32.784509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.784318 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" event={"ID":"f6ca7467-f49f-4151-85fc-5fc490e9a518","Type":"ContainerDied","Data":"01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c"} Nov 03 21:16:32.784509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.784330 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" Nov 03 21:16:32.784509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.784358 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr" event={"ID":"f6ca7467-f49f-4151-85fc-5fc490e9a518","Type":"ContainerDied","Data":"54a1124f04a25f2d137ac3a9db3e6a2d2be0043144e267ad48d756c5891d88a2"} Nov 03 21:16:32.784509 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.784378 4227 scope.go:122] "RemoveContainer" containerID="01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c" Nov 03 21:16:32.793930 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.793914 4227 scope.go:122] "RemoveContainer" containerID="b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5" Nov 03 21:16:32.801709 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.801690 4227 scope.go:122] "RemoveContainer" containerID="01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c" Nov 03 21:16:32.801920 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:32.801901 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c\": container with ID starting with 01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c not found: ID does not exist" containerID="01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c" Nov 03 21:16:32.801967 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.801931 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c"} err="failed to get container status \"01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c\": rpc error: code = NotFound desc = could not find container \"01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c\": container with ID starting with 01b131698c8291fb821959dce525d654fd1ba11f45b2b038e202d3b1d3251b0c not found: ID does not exist" Nov 03 21:16:32.801967 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.801956 4227 scope.go:122] "RemoveContainer" containerID="b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5" Nov 03 21:16:32.802216 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:32.802194 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5\": container with ID starting with b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5 not found: ID does not exist" containerID="b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5" Nov 03 21:16:32.802332 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.802219 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5"} err="failed to get container status \"b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5\": rpc error: code = NotFound desc = could not find container \"b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5\": container with ID starting with b43cda583e8d4553a30b6a2e0119ccfe058746b39e99679051336e0ae94db3b5 not found: ID does not exist" Nov 03 21:16:32.984747 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:32.984714 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr"] Nov 03 21:16:33.031215 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:33.031184 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-562eb2-openshift-default-ccb795d89-cnvzr"] Nov 03 21:16:34.077918 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.077895 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:16:34.118366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118288 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-certs\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118326 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-credential-socket\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118374 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-envoy\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118407 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-token\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-token\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118457 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-data\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118489 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-podinfo\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118525 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-socket\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118571 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-certs" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.118821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118577 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6278553-4b48-4f8c-bbc3-d3217718265c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6278553-4b48-4f8c-bbc3-d3217718265c-istiod-ca-cert\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118655 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-credential-socket" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.118821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118710 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-kube-api-access-9x6pn\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-kube-api-access-9x6pn\") pod \"f6278553-4b48-4f8c-bbc3-d3217718265c\" (UID: \"f6278553-4b48-4f8c-bbc3-d3217718265c\") " Nov 03 21:16:34.118821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118814 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-data" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.119078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118912 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-socket" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.119078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.118931 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6278553-4b48-4f8c-bbc3-d3217718265c-istiod-ca-cert" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:34.119078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.119053 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.119078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.119067 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.119078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.119077 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6278553-4b48-4f8c-bbc3-d3217718265c-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.119385 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.119086 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.119385 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.119099 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.120629 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.120602 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-envoy" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.120744 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.120652 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-token" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:34.120744 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.120691 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-podinfo" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:16:34.120744 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.120732 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-kube-api-access-9x6pn" pod "f6278553-4b48-4f8c-bbc3-d3217718265c" (UID: "f6278553-4b48-4f8c-bbc3-d3217718265c"). InnerVolumeSpecName "kube-api-access-9x6pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:34.219669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.219632 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.219669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.219658 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.219669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.219669 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6278553-4b48-4f8c-bbc3-d3217718265c-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.219922 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.219682 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6pn\" (UniqueName: \"kubernetes.io/projected/f6278553-4b48-4f8c-bbc3-d3217718265c-kube-api-access-9x6pn\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.282349 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.282317 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq"] Nov 03 21:16:34.797777 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.797739 4227 generic.go:356] "Generic (PLEG): container finished" podID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerID="302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0" exitCode=0 Nov 03 21:16:34.797957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.797793 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" Nov 03 21:16:34.797957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.797816 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" event={"ID":"f6278553-4b48-4f8c-bbc3-d3217718265c","Type":"ContainerDied","Data":"302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0"} Nov 03 21:16:34.797957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.797875 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t" event={"ID":"f6278553-4b48-4f8c-bbc3-d3217718265c","Type":"ContainerDied","Data":"2253aa93da44d16739e0b9cb18111bb4772d116b46b1f42bbbc5396badda6ed6"} Nov 03 21:16:34.797957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.797906 4227 scope.go:122] "RemoveContainer" containerID="302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0" Nov 03 21:16:34.807813 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.807791 4227 scope.go:122] "RemoveContainer" containerID="7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f" Nov 03 21:16:34.817815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.817370 4227 scope.go:122] "RemoveContainer" containerID="302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0" Nov 03 21:16:34.817815 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:34.817760 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0\": container with ID starting with 302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0 not found: ID does not exist" containerID="302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0" Nov 03 21:16:34.817815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.817796 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0"} err="failed to get container status \"302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0\": rpc error: code = NotFound desc = could not find container \"302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0\": container with ID starting with 302c61d07c1158dd6a7ba8f3cb191ed9fcaf3c22a6236e831f6e076687d913f0 not found: ID does not exist" Nov 03 21:16:34.817975 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.817820 4227 scope.go:122] "RemoveContainer" containerID="7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f" Nov 03 21:16:34.818657 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:34.818634 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f\": container with ID starting with 7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f not found: ID does not exist" containerID="7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f" Nov 03 21:16:34.818754 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.818666 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f"} err="failed to get container status \"7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f\": rpc error: code = NotFound desc = could not find container \"7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f\": container with ID starting with 7a0e7420f997c575a75c933df406e29f1bf35bd53cf8e629492be43d98f3732f not found: ID does not exist" Nov 03 21:16:34.899358 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.899337 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" path="/var/lib/kubelet/pods/f6ca7467-f49f-4151-85fc-5fc490e9a518/volumes" Nov 03 21:16:34.950607 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.950578 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t"] Nov 03 21:16:34.997079 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:34.997049 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-98d1d2-openshift-default-757c4d8785-n4x9t"] Nov 03 21:16:36.898479 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:36.898446 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" path="/var/lib/kubelet/pods/f6278553-4b48-4f8c-bbc3-d3217718265c/volumes" Nov 03 21:16:49.301799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.301744 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" containerID="cri-o://7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621" gracePeriod=45 Nov 03 21:16:49.552063 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.552005 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 21:16:49.690664 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.690634 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86280466-9e84-42a4-a0be-9798a081c9cd-kube-api-access-lkcn5\" (UniqueName: \"kubernetes.io/projected/86280466-9e84-42a4-a0be-9798a081c9cd-kube-api-access-lkcn5\") pod \"86280466-9e84-42a4-a0be-9798a081c9cd\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " Nov 03 21:16:49.690840 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.690689 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86280466-9e84-42a4-a0be-9798a081c9cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/86280466-9e84-42a4-a0be-9798a081c9cd-tls-certs\") pod \"86280466-9e84-42a4-a0be-9798a081c9cd\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " Nov 03 21:16:49.690840 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.690732 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-model-cache\") pod \"86280466-9e84-42a4-a0be-9798a081c9cd\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " Nov 03 21:16:49.690840 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.690805 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-home\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-home\") pod \"86280466-9e84-42a4-a0be-9798a081c9cd\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " Nov 03 21:16:49.691024 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.690850 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-tmp-dir\") pod \"86280466-9e84-42a4-a0be-9798a081c9cd\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " Nov 03 21:16:49.691024 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.690914 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-dshm\") pod \"86280466-9e84-42a4-a0be-9798a081c9cd\" (UID: \"86280466-9e84-42a4-a0be-9798a081c9cd\") " Nov 03 21:16:49.691143 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.691079 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-model-cache" pod "86280466-9e84-42a4-a0be-9798a081c9cd" (UID: "86280466-9e84-42a4-a0be-9798a081c9cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.691295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.691275 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.691416 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.691352 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-tmp-dir" pod "86280466-9e84-42a4-a0be-9798a081c9cd" (UID: "86280466-9e84-42a4-a0be-9798a081c9cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.691729 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.691703 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-home" pod "86280466-9e84-42a4-a0be-9798a081c9cd" (UID: "86280466-9e84-42a4-a0be-9798a081c9cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.693212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.693179 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86280466-9e84-42a4-a0be-9798a081c9cd-kube-api-access-lkcn5" pod "86280466-9e84-42a4-a0be-9798a081c9cd" (UID: "86280466-9e84-42a4-a0be-9798a081c9cd"). InnerVolumeSpecName "kube-api-access-lkcn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:49.693319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.693282 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86280466-9e84-42a4-a0be-9798a081c9cd-tls-certs" pod "86280466-9e84-42a4-a0be-9798a081c9cd" (UID: "86280466-9e84-42a4-a0be-9798a081c9cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:49.693319 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.693286 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-dshm" pod "86280466-9e84-42a4-a0be-9798a081c9cd" (UID: "86280466-9e84-42a4-a0be-9798a081c9cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:49.792321 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.792285 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.792321 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.792324 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcn5\" (UniqueName: \"kubernetes.io/projected/86280466-9e84-42a4-a0be-9798a081c9cd-kube-api-access-lkcn5\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.792527 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.792340 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86280466-9e84-42a4-a0be-9798a081c9cd-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.792527 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.792354 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.792527 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.792367 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86280466-9e84-42a4-a0be-9798a081c9cd-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:49.881041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.880958 4227 generic.go:356] "Generic (PLEG): container finished" podID="86280466-9e84-42a4-a0be-9798a081c9cd" containerID="7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621" exitCode=0 Nov 03 21:16:49.881041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.881000 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" Nov 03 21:16:49.881242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.881040 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" event={"ID":"86280466-9e84-42a4-a0be-9798a081c9cd","Type":"ContainerDied","Data":"7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621"} Nov 03 21:16:49.881242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.881086 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq" event={"ID":"86280466-9e84-42a4-a0be-9798a081c9cd","Type":"ContainerDied","Data":"df03275a3c3076ddf15a7f9805e2b6b9634afda23806bfb4ae33b7b26f0fda9d"} Nov 03 21:16:49.881242 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.881108 4227 scope.go:122] "RemoveContainer" containerID="7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621" Nov 03 21:16:49.890394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.890375 4227 scope.go:122] "RemoveContainer" containerID="7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621" Nov 03 21:16:49.890708 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:49.890686 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621\": container with ID starting with 7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621 not found: ID does not exist" containerID="7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621" Nov 03 21:16:49.890780 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:49.890713 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621"} err="failed to get container status \"7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621\": rpc error: code = NotFound desc = could not find container \"7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621\": container with ID starting with 7fc6424503742ebaf162a76923b9d52d7b07bdace068fa70d05cb1e4ecdab621 not found: ID does not exist" Nov 03 21:16:50.032541 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:50.032509 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq"] Nov 03 21:16:50.079672 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:50.079631 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/distinct-w4-984cbc-kserve-8fd4c85c8-cwhtq"] Nov 03 21:16:50.901344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:50.901307 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" path="/var/lib/kubelet/pods/86280466-9e84-42a4-a0be-9798a081c9cd/volumes" Nov 03 21:16:56.086178 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:16:56.086124 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:16:59.448233 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:59.448191 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx"] Nov 03 21:16:59.448832 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:16:59.448457 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" containerID="cri-o://b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae" gracePeriod=30 Nov 03 21:17:01.460825 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:01.460784 4227 patch_prober.go:29] interesting pod/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:17:01.461288 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:01.460849 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:17:04.709279 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.709253 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 21:17:04.766452 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766379 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-kube-api-access-8n4lx\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-kube-api-access-8n4lx\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766452 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766437 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-credential-socket\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766654 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766473 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-podinfo\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766654 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766501 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-envoy\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766654 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766572 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-data\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766654 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766606 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-socket\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766867 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766690 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-certs\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766867 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766736 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-token\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-token\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.766867 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766783 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-credential-socket" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:04.766867 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766803 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb79187f-8c1c-4532-8006-f011d3c72e5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb79187f-8c1c-4532-8006-f011d3c72e5e-istiod-ca-cert\") pod \"cb79187f-8c1c-4532-8006-f011d3c72e5e\" (UID: \"cb79187f-8c1c-4532-8006-f011d3c72e5e\") " Nov 03 21:17:04.767082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.766965 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-certs" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:04.767082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.767012 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-socket" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:04.767207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.767090 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-data" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:04.767283 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.767263 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.767342 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.767288 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.767342 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.767303 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.767342 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.767311 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb79187f-8c1c-4532-8006-f011d3c72e5e-istiod-ca-cert" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:04.767342 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.767320 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.768880 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.768858 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-token" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:04.769012 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.768996 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-podinfo" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:04.769185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.769148 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-kube-api-access-8n4lx" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "kube-api-access-8n4lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:04.769506 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.769475 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-envoy" pod "cb79187f-8c1c-4532-8006-f011d3c72e5e" (UID: "cb79187f-8c1c-4532-8006-f011d3c72e5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:04.868947 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.868915 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb79187f-8c1c-4532-8006-f011d3c72e5e-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.868947 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.868939 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4lx\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-kube-api-access-8n4lx\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.869112 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.868954 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.869112 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.868967 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.869112 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.868976 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79187f-8c1c-4532-8006-f011d3c72e5e-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:04.973521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.973485 4227 generic.go:356] "Generic (PLEG): container finished" podID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerID="b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae" exitCode=0 Nov 03 21:17:04.973669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.973528 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" Nov 03 21:17:04.973669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.973560 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" event={"ID":"cb79187f-8c1c-4532-8006-f011d3c72e5e","Type":"ContainerDied","Data":"b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae"} Nov 03 21:17:04.973669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.973616 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx" event={"ID":"cb79187f-8c1c-4532-8006-f011d3c72e5e","Type":"ContainerDied","Data":"6d715b882467b0e2d21afdd579d6eb3cbdbbc111715f538e9849a8669ef94e30"} Nov 03 21:17:04.973669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.973645 4227 scope.go:122] "RemoveContainer" containerID="b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae" Nov 03 21:17:04.983862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.983830 4227 scope.go:122] "RemoveContainer" containerID="981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102" Nov 03 21:17:04.991514 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.991485 4227 scope.go:122] "RemoveContainer" containerID="b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae" Nov 03 21:17:04.991786 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:04.991767 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae\": container with ID starting with b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae not found: ID does not exist" containerID="b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae" Nov 03 21:17:04.991859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.991793 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae"} err="failed to get container status \"b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae\": rpc error: code = NotFound desc = could not find container \"b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae\": container with ID starting with b4934288eae0e2a71669be6a4fb8667bb4eb559b0fffcd9f07367d592bbc4bae not found: ID does not exist" Nov 03 21:17:04.991859 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.991811 4227 scope.go:122] "RemoveContainer" containerID="981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102" Nov 03 21:17:04.992030 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:04.992015 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102\": container with ID starting with 981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102 not found: ID does not exist" containerID="981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102" Nov 03 21:17:04.992088 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:04.992033 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102"} err="failed to get container status \"981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102\": rpc error: code = NotFound desc = could not find container \"981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102\": container with ID starting with 981ac0c9361e9b2f75daf0cf41edf256c6d37e036639047d6f9f4ddc0c668102 not found: ID does not exist" Nov 03 21:17:05.128382 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:05.128309 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx"] Nov 03 21:17:05.132865 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:05.132844 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-45f792-openshift-default-74884d7bd7-v56tx"] Nov 03 21:17:06.898678 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:06.898647 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" path="/var/lib/kubelet/pods/cb79187f-8c1c-4532-8006-f011d3c72e5e/volumes" Nov 03 21:17:11.987444 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:11.987406 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z"] Nov 03 21:17:11.987864 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:11.987628 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" podUID="97e0ffb6-e265-40b3-9e31-6dfe02ee743d" containerName="maas-api" containerID="cri-o://6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529" gracePeriod=30 Nov 03 21:17:12.301687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.301663 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 21:17:12.343590 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.343557 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-openshift-service-ca\") pod \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " Nov 03 21:17:12.343797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.343612 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-kube-api-access-6l5sd\" (UniqueName: \"kubernetes.io/projected/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-kube-api-access-6l5sd\") pod \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " Nov 03 21:17:12.343797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.343665 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-maas-api-tls\") pod \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " Nov 03 21:17:12.343797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.343696 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-metrics-certs\") pod \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\" (UID: \"97e0ffb6-e265-40b3-9e31-6dfe02ee743d\") " Nov 03 21:17:12.343960 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.343937 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-openshift-service-ca" pod "97e0ffb6-e265-40b3-9e31-6dfe02ee743d" (UID: "97e0ffb6-e265-40b3-9e31-6dfe02ee743d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:12.344629 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.344180 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:12.346096 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.346065 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-metrics-certs" pod "97e0ffb6-e265-40b3-9e31-6dfe02ee743d" (UID: "97e0ffb6-e265-40b3-9e31-6dfe02ee743d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:12.346331 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.346298 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-kube-api-access-6l5sd" pod "97e0ffb6-e265-40b3-9e31-6dfe02ee743d" (UID: "97e0ffb6-e265-40b3-9e31-6dfe02ee743d"). InnerVolumeSpecName "kube-api-access-6l5sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:12.346695 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.346670 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-maas-api-tls" pod "97e0ffb6-e265-40b3-9e31-6dfe02ee743d" (UID: "97e0ffb6-e265-40b3-9e31-6dfe02ee743d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:12.445469 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.445438 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l5sd\" (UniqueName: \"kubernetes.io/projected/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-kube-api-access-6l5sd\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:12.445469 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.445470 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:12.445657 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:12.445485 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e0ffb6-e265-40b3-9e31-6dfe02ee743d-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:13.013750 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.013711 4227 generic.go:356] "Generic (PLEG): container finished" podID="97e0ffb6-e265-40b3-9e31-6dfe02ee743d" containerID="6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529" exitCode=0 Nov 03 21:17:13.014262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.013793 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" Nov 03 21:17:13.014262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.013814 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" event={"ID":"97e0ffb6-e265-40b3-9e31-6dfe02ee743d","Type":"ContainerDied","Data":"6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529"} Nov 03 21:17:13.014262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.013878 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z" event={"ID":"97e0ffb6-e265-40b3-9e31-6dfe02ee743d","Type":"ContainerDied","Data":"3c76dc650a7a555747d1ffc0bd17548f22cf4e009cdea84771cfcba1cfe5737e"} Nov 03 21:17:13.014262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.013907 4227 scope.go:122] "RemoveContainer" containerID="6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529" Nov 03 21:17:13.024013 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.023997 4227 scope.go:122] "RemoveContainer" containerID="6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529" Nov 03 21:17:13.024290 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:13.024271 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529\": container with ID starting with 6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529 not found: ID does not exist" containerID="6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529" Nov 03 21:17:13.024334 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.024299 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529"} err="failed to get container status \"6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529\": rpc error: code = NotFound desc = could not find container \"6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529\": container with ID starting with 6dc9fb353a6e34c0c20c08f409cb9fe2263300719f76ab5dadd5a7b95655f529 not found: ID does not exist" Nov 03 21:17:13.167443 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.167366 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z"] Nov 03 21:17:13.216596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:13.215443 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-984cbc-5d96847f74-trz2z"] Nov 03 21:17:14.899742 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:14.899708 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e0ffb6-e265-40b3-9e31-6dfe02ee743d" path="/var/lib/kubelet/pods/97e0ffb6-e265-40b3-9e31-6dfe02ee743d/volumes" Nov 03 21:17:23.019264 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.019229 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp"] Nov 03 21:17:23.019725 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.019416 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" podUID="ee287795-b1a2-4763-8eec-f7ef207e277b" containerName="maas-api" containerID="cri-o://c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3" gracePeriod=30 Nov 03 21:17:23.292357 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.292332 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 21:17:23.368358 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.368333 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-metrics-certs\") pod \"ee287795-b1a2-4763-8eec-f7ef207e277b\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " Nov 03 21:17:23.368503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.368387 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee287795-b1a2-4763-8eec-f7ef207e277b-kube-api-access-95n6q\" (UniqueName: \"kubernetes.io/projected/ee287795-b1a2-4763-8eec-f7ef207e277b-kube-api-access-95n6q\") pod \"ee287795-b1a2-4763-8eec-f7ef207e277b\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " Nov 03 21:17:23.368503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.368457 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee287795-b1a2-4763-8eec-f7ef207e277b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee287795-b1a2-4763-8eec-f7ef207e277b-openshift-service-ca\") pod \"ee287795-b1a2-4763-8eec-f7ef207e277b\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " Nov 03 21:17:23.368581 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.368520 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls\") pod \"ee287795-b1a2-4763-8eec-f7ef207e277b\" (UID: \"ee287795-b1a2-4763-8eec-f7ef207e277b\") " Nov 03 21:17:23.368865 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.368839 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee287795-b1a2-4763-8eec-f7ef207e277b-openshift-service-ca" pod "ee287795-b1a2-4763-8eec-f7ef207e277b" (UID: "ee287795-b1a2-4763-8eec-f7ef207e277b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:23.370391 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.370370 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-metrics-certs" pod "ee287795-b1a2-4763-8eec-f7ef207e277b" (UID: "ee287795-b1a2-4763-8eec-f7ef207e277b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:23.370584 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.370568 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls" pod "ee287795-b1a2-4763-8eec-f7ef207e277b" (UID: "ee287795-b1a2-4763-8eec-f7ef207e277b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:23.370631 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.370597 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee287795-b1a2-4763-8eec-f7ef207e277b-kube-api-access-95n6q" pod "ee287795-b1a2-4763-8eec-f7ef207e277b" (UID: "ee287795-b1a2-4763-8eec-f7ef207e277b"). InnerVolumeSpecName "kube-api-access-95n6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:23.469978 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.469945 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee287795-b1a2-4763-8eec-f7ef207e277b-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:23.469978 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.469972 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:23.469978 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.469981 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee287795-b1a2-4763-8eec-f7ef207e277b-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:23.470231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:23.469990 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95n6q\" (UniqueName: \"kubernetes.io/projected/ee287795-b1a2-4763-8eec-f7ef207e277b-kube-api-access-95n6q\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:24.065243 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.065212 4227 generic.go:356] "Generic (PLEG): container finished" podID="ee287795-b1a2-4763-8eec-f7ef207e277b" containerID="c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3" exitCode=0 Nov 03 21:17:24.065716 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.065255 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" Nov 03 21:17:24.065716 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.065280 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" event={"ID":"ee287795-b1a2-4763-8eec-f7ef207e277b","Type":"ContainerDied","Data":"c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3"} Nov 03 21:17:24.065716 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.065323 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp" event={"ID":"ee287795-b1a2-4763-8eec-f7ef207e277b","Type":"ContainerDied","Data":"94aff520f5e16168c93857ed6b046ccdc5e4e27e03a1c7afabeff8330f309880"} Nov 03 21:17:24.065716 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.065344 4227 scope.go:122] "RemoveContainer" containerID="c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3" Nov 03 21:17:24.074944 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.074926 4227 scope.go:122] "RemoveContainer" containerID="c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3" Nov 03 21:17:24.075222 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:24.075202 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3\": container with ID starting with c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3 not found: ID does not exist" containerID="c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3" Nov 03 21:17:24.075287 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.075231 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3"} err="failed to get container status \"c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3\": rpc error: code = NotFound desc = could not find container \"c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3\": container with ID starting with c1de760647f03908a3eed237b57af80454db4a701964f2beafdab7fdcb1c8cf3 not found: ID does not exist" Nov 03 21:17:24.217971 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.217941 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp"] Nov 03 21:17:24.264774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.264741 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e862b-556d558fd-vf6lp"] Nov 03 21:17:24.898840 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:24.898797 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee287795-b1a2-4763-8eec-f7ef207e277b" path="/var/lib/kubelet/pods/ee287795-b1a2-4763-8eec-f7ef207e277b/volumes" Nov 03 21:17:26.082019 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:26.081981 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:17:37.044057 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:37.043980 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l"] Nov 03 21:17:37.090950 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:37.090920 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492"] Nov 03 21:17:37.093765 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:37.093725 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c"] Nov 03 21:17:44.172393 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172358 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl"] Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172709 4227 cpu_manager.go:414] "containerMap: removing container" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172721 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172730 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172735 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172749 4227 cpu_manager.go:414] "containerMap: removing container" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172757 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172767 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172772 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172782 4227 cpu_manager.go:414] "containerMap: removing container" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172786 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172796 4227 cpu_manager.go:414] "containerMap: removing container" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172800 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172806 4227 cpu_manager.go:414] "containerMap: removing container" podUID="97e0ffb6-e265-40b3-9e31-6dfe02ee743d" containerName="maas-api" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172811 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e0ffb6-e265-40b3-9e31-6dfe02ee743d" containerName="maas-api" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172817 4227 cpu_manager.go:414] "containerMap: removing container" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172823 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172831 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ee287795-b1a2-4763-8eec-f7ef207e277b" containerName="maas-api" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172835 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee287795-b1a2-4763-8eec-f7ef207e277b" containerName="maas-api" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172843 4227 cpu_manager.go:414] "containerMap: removing container" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172847 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172852 4227 cpu_manager.go:414] "containerMap: removing container" podUID="da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" containerName="payload-processing" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172856 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" containerName="payload-processing" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172862 4227 cpu_manager.go:414] "containerMap: removing container" podUID="bd24478d-4d06-4727-a7e9-7b392025bd96" containerName="payload-pre-processing" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172866 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd24478d-4d06-4727-a7e9-7b392025bd96" containerName="payload-pre-processing" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172874 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f15418ca-ed1c-46f7-94d7-3e0d58375ca7" containerName="main" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172878 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15418ca-ed1c-46f7-94d7-3e0d58375ca7" containerName="main" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172885 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172890 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172896 4227 cpu_manager.go:414] "containerMap: removing container" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172901 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172907 4227 cpu_manager.go:414] "containerMap: removing container" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172911 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172918 4227 cpu_manager.go:414] "containerMap: removing container" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172923 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172929 4227 cpu_manager.go:414] "containerMap: removing container" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172934 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172939 4227 cpu_manager.go:414] "containerMap: removing container" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.172917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172944 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172952 4227 cpu_manager.go:414] "containerMap: removing container" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172957 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172963 4227 cpu_manager.go:414] "containerMap: removing container" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172968 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172979 4227 cpu_manager.go:414] "containerMap: removing container" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.172984 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173034 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173041 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="86280466-9e84-42a4-a0be-9798a081c9cd" containerName="main" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173048 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173054 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173060 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173065 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173071 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173077 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e0ffb6-e265-40b3-9e31-6dfe02ee743d" containerName="maas-api" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173082 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173087 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee287795-b1a2-4763-8eec-f7ef207e277b" containerName="maas-api" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173093 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="93102cbf-09e6-4941-b8e6-d59cc01d8bee" containerName="istio-proxy" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173098 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7637d2-a8a2-4b8d-81e2-6d6af3f71b8c" containerName="payload-processing" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173104 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd24478d-4d06-4727-a7e9-7b392025bd96" containerName="payload-pre-processing" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173108 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15418ca-ed1c-46f7-94d7-3e0d58375ca7" containerName="main" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173114 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ca7467-f49f-4151-85fc-5fc490e9a518" containerName="istio-proxy" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173119 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb79187f-8c1c-4532-8006-f011d3c72e5e" containerName="istio-proxy" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173123 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173127 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173132 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.174946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.173138 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b67219-a600-4166-846f-8b89475bfd36" containerName="cleanup" Nov 03 21:17:44.177537 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.177515 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.228449 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.228425 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl"] Nov 03 21:17:44.269012 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.268971 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.269212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.269035 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ptz\" (UniqueName: \"kubernetes.io/projected/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-kube-api-access-l4ptz\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.269212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.269102 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.272679 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.272657 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:17:44.272777 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.272719 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:17:44.272777 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.272730 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:17:44.272863 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.272793 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:17:44.370216 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.370153 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ptz\" (UniqueName: \"kubernetes.io/projected/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-kube-api-access-l4ptz\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.370381 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.370230 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.370381 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.370363 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.370986 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.370959 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.372512 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.372494 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.422689 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.422630 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ptz\" (UniqueName: \"kubernetes.io/projected/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-kube-api-access-l4ptz\") pod \"maas-api-revoke-sub-86eb7c8b-f4mkl\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.492973 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.492948 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:44.861043 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:44.860965 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl"] Nov 03 21:17:44.868137 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:17:44.868111 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0ba26d_e248_4f02_a7cf_58121f1b9d1d.slice/crio-6ccb1edc8b325b9a2cb5bb4e96a4572ccf40c0fdaee1ce25df19dcb33efc47d5 WatchSource:0}: Error finding container 6ccb1edc8b325b9a2cb5bb4e96a4572ccf40c0fdaee1ce25df19dcb33efc47d5: Status 404 returned error can't find the container with id 6ccb1edc8b325b9a2cb5bb4e96a4572ccf40c0fdaee1ce25df19dcb33efc47d5 Nov 03 21:17:45.168428 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:45.168390 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" event={"ID":"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d","Type":"ContainerStarted","Data":"d8e9eafb146b92e008af0b625bcd51fde9c2e560b3a0a936fbef2d9e3c1d76ef"} Nov 03 21:17:45.168564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:45.168445 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" event={"ID":"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d","Type":"ContainerStarted","Data":"6ccb1edc8b325b9a2cb5bb4e96a4572ccf40c0fdaee1ce25df19dcb33efc47d5"} Nov 03 21:17:45.274683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:45.274632 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" podStartSLOduration=1.274617279 podStartE2EDuration="1.274617279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:45.273030196 +0000 UTC m=+2932.510008070" watchObservedRunningTime="2026-11-03 21:17:45.274617279 +0000 UTC m=+2932.511595153" Nov 03 21:17:46.173899 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:46.173865 4227 generic.go:356] "Generic (PLEG): container finished" podID="6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" containerID="d8e9eafb146b92e008af0b625bcd51fde9c2e560b3a0a936fbef2d9e3c1d76ef" exitCode=0 Nov 03 21:17:46.174109 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:46.173957 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" event={"ID":"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d","Type":"ContainerDied","Data":"d8e9eafb146b92e008af0b625bcd51fde9c2e560b3a0a936fbef2d9e3c1d76ef"} Nov 03 21:17:47.344747 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.344723 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:47.420494 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.420463 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-xks-ca\") pod \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " Nov 03 21:17:47.420658 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.420616 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-openshift-ca\") pod \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " Nov 03 21:17:47.420732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.420667 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-kube-api-access-l4ptz\" (UniqueName: \"kubernetes.io/projected/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-kube-api-access-l4ptz\") pod \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\" (UID: \"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d\") " Nov 03 21:17:47.420928 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.420906 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-openshift-ca" pod "6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" (UID: "6f0ba26d-e248-4f02-a7cf-58121f1b9d1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:47.422616 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.422592 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-xks-ca" pod "6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" (UID: "6f0ba26d-e248-4f02-a7cf-58121f1b9d1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:47.422712 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.422635 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-kube-api-access-l4ptz" pod "6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" (UID: "6f0ba26d-e248-4f02-a7cf-58121f1b9d1d"). InnerVolumeSpecName "kube-api-access-l4ptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:47.521405 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.521335 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-openshift-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:47.521405 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.521360 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4ptz\" (UniqueName: \"kubernetes.io/projected/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-kube-api-access-l4ptz\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:47.521405 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:47.521371 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d-maas-api-xks-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:48.185211 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:48.185186 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" Nov 03 21:17:48.185360 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:48.185176 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl" event={"ID":"6f0ba26d-e248-4f02-a7cf-58121f1b9d1d","Type":"ContainerDied","Data":"6ccb1edc8b325b9a2cb5bb4e96a4572ccf40c0fdaee1ce25df19dcb33efc47d5"} Nov 03 21:17:48.185360 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:48.185342 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ccb1edc8b325b9a2cb5bb4e96a4572ccf40c0fdaee1ce25df19dcb33efc47d5" Nov 03 21:17:52.063556 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.063517 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" containerID="cri-o://e34495afed11661bd43e64b11f49ef924462471c97622c6c10bafba9ba4d9538" gracePeriod=45 Nov 03 21:17:52.111436 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.111408 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" containerID="cri-o://44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67" gracePeriod=45 Nov 03 21:17:52.113083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.113055 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" containerID="cri-o://53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675" gracePeriod=45 Nov 03 21:17:52.151882 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.151852 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": dial tcp 10.134.0.40:8000: connect: connection refused" Nov 03 21:17:52.204624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.204587 4227 generic.go:356] "Generic (PLEG): container finished" podID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerID="e34495afed11661bd43e64b11f49ef924462471c97622c6c10bafba9ba4d9538" exitCode=0 Nov 03 21:17:52.204803 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.204633 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" event={"ID":"0352c748-5879-44cc-a69a-8fa2f1eedffa","Type":"ContainerDied","Data":"e34495afed11661bd43e64b11f49ef924462471c97622c6c10bafba9ba4d9538"} Nov 03 21:17:52.380552 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.380525 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 21:17:52.396966 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.396937 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 21:17:52.400287 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.400272 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 21:17:52.470863 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.470836 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-home\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-home\") pod \"0352c748-5879-44cc-a69a-8fa2f1eedffa\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " Nov 03 21:17:52.471041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.470872 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tmp-dir\") pod \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " Nov 03 21:17:52.471041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.470897 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-model-cache\") pod \"f781fc90-550e-4649-b618-a1375b3cfbb8\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " Nov 03 21:17:52.471041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.470964 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-home\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-home\") pod \"f781fc90-550e-4649-b618-a1375b3cfbb8\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " Nov 03 21:17:52.471041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.470997 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0352c748-5879-44cc-a69a-8fa2f1eedffa-tls-certs\" (UniqueName: \"kubernetes.io/secret/0352c748-5879-44cc-a69a-8fa2f1eedffa-tls-certs\") pod \"0352c748-5879-44cc-a69a-8fa2f1eedffa\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " Nov 03 21:17:52.471041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471025 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-home\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-home\") pod \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471057 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-model-cache\") pod \"0352c748-5879-44cc-a69a-8fa2f1eedffa\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471119 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f781fc90-550e-4649-b618-a1375b3cfbb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f781fc90-550e-4649-b618-a1375b3cfbb8-tls-certs\") pod \"f781fc90-550e-4649-b618-a1375b3cfbb8\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471141 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-home" pod "0352c748-5879-44cc-a69a-8fa2f1eedffa" (UID: "0352c748-5879-44cc-a69a-8fa2f1eedffa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471148 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-model-cache\") pod \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471148 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tmp-dir" pod "e5fe195c-3ab8-4197-ae4d-5c3ca2452117" (UID: "e5fe195c-3ab8-4197-ae4d-5c3ca2452117"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471224 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-dshm\") pod \"f781fc90-550e-4649-b618-a1375b3cfbb8\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471248 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-model-cache" pod "f781fc90-550e-4649-b618-a1375b3cfbb8" (UID: "f781fc90-550e-4649-b618-a1375b3cfbb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471257 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-kube-api-access-9t6v6\" (UniqueName: \"kubernetes.io/projected/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-kube-api-access-9t6v6\") pod \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " Nov 03 21:17:52.471344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471312 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-dshm\") pod \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471375 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-tmp-dir\") pod \"f781fc90-550e-4649-b618-a1375b3cfbb8\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471402 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-model-cache" pod "0352c748-5879-44cc-a69a-8fa2f1eedffa" (UID: "0352c748-5879-44cc-a69a-8fa2f1eedffa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471423 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f781fc90-550e-4649-b618-a1375b3cfbb8-kube-api-access-sx2vl\" (UniqueName: \"kubernetes.io/projected/f781fc90-550e-4649-b618-a1375b3cfbb8-kube-api-access-sx2vl\") pod \"f781fc90-550e-4649-b618-a1375b3cfbb8\" (UID: \"f781fc90-550e-4649-b618-a1375b3cfbb8\") " Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471475 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tls-certs\") pod \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\" (UID: \"e5fe195c-3ab8-4197-ae4d-5c3ca2452117\") " Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471504 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-tmp-dir\") pod \"0352c748-5879-44cc-a69a-8fa2f1eedffa\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471543 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-dshm\") pod \"0352c748-5879-44cc-a69a-8fa2f1eedffa\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471581 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0352c748-5879-44cc-a69a-8fa2f1eedffa-kube-api-access-c956f\" (UniqueName: \"kubernetes.io/projected/0352c748-5879-44cc-a69a-8fa2f1eedffa-kube-api-access-c956f\") pod \"0352c748-5879-44cc-a69a-8fa2f1eedffa\" (UID: \"0352c748-5879-44cc-a69a-8fa2f1eedffa\") " Nov 03 21:17:52.471806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471603 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-home" pod "f781fc90-550e-4649-b618-a1375b3cfbb8" (UID: "f781fc90-550e-4649-b618-a1375b3cfbb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471861 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-model-cache" pod "e5fe195c-3ab8-4197-ae4d-5c3ca2452117" (UID: "e5fe195c-3ab8-4197-ae4d-5c3ca2452117"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471884 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-home" pod "e5fe195c-3ab8-4197-ae4d-5c3ca2452117" (UID: "e5fe195c-3ab8-4197-ae4d-5c3ca2452117"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.471982 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472000 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472067 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472084 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472098 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472102 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-tmp-dir" pod "f781fc90-550e-4649-b618-a1375b3cfbb8" (UID: "f781fc90-550e-4649-b618-a1375b3cfbb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472112 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.472271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472143 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.472802 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.472347 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-tmp-dir" pod "0352c748-5879-44cc-a69a-8fa2f1eedffa" (UID: "0352c748-5879-44cc-a69a-8fa2f1eedffa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.474379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.474350 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0352c748-5879-44cc-a69a-8fa2f1eedffa-tls-certs" pod "0352c748-5879-44cc-a69a-8fa2f1eedffa" (UID: "0352c748-5879-44cc-a69a-8fa2f1eedffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:52.474379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.474375 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-kube-api-access-9t6v6" pod "e5fe195c-3ab8-4197-ae4d-5c3ca2452117" (UID: "e5fe195c-3ab8-4197-ae4d-5c3ca2452117"). InnerVolumeSpecName "kube-api-access-9t6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:52.474580 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.474450 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-dshm" pod "e5fe195c-3ab8-4197-ae4d-5c3ca2452117" (UID: "e5fe195c-3ab8-4197-ae4d-5c3ca2452117"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.474647 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.474631 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0352c748-5879-44cc-a69a-8fa2f1eedffa-kube-api-access-c956f" pod "0352c748-5879-44cc-a69a-8fa2f1eedffa" (UID: "0352c748-5879-44cc-a69a-8fa2f1eedffa"). InnerVolumeSpecName "kube-api-access-c956f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:52.476004 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.475977 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tls-certs" pod "e5fe195c-3ab8-4197-ae4d-5c3ca2452117" (UID: "e5fe195c-3ab8-4197-ae4d-5c3ca2452117"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:52.476237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.476200 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f781fc90-550e-4649-b618-a1375b3cfbb8-kube-api-access-sx2vl" pod "f781fc90-550e-4649-b618-a1375b3cfbb8" (UID: "f781fc90-550e-4649-b618-a1375b3cfbb8"). InnerVolumeSpecName "kube-api-access-sx2vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:52.476353 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.476280 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-dshm" pod "0352c748-5879-44cc-a69a-8fa2f1eedffa" (UID: "0352c748-5879-44cc-a69a-8fa2f1eedffa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.476398 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.476382 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f781fc90-550e-4649-b618-a1375b3cfbb8-tls-certs" pod "f781fc90-550e-4649-b618-a1375b3cfbb8" (UID: "f781fc90-550e-4649-b618-a1375b3cfbb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:52.476579 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.476562 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-dshm" pod "f781fc90-550e-4649-b618-a1375b3cfbb8" (UID: "f781fc90-550e-4649-b618-a1375b3cfbb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:52.573285 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573251 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573285 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573288 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2vl\" (UniqueName: \"kubernetes.io/projected/f781fc90-550e-4649-b618-a1375b3cfbb8-kube-api-access-sx2vl\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573304 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573320 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573333 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352c748-5879-44cc-a69a-8fa2f1eedffa-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573346 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c956f\" (UniqueName: \"kubernetes.io/projected/0352c748-5879-44cc-a69a-8fa2f1eedffa-kube-api-access-c956f\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573361 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352c748-5879-44cc-a69a-8fa2f1eedffa-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573375 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f781fc90-550e-4649-b618-a1375b3cfbb8-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573386 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f781fc90-550e-4649-b618-a1375b3cfbb8-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573399 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6v6\" (UniqueName: \"kubernetes.io/projected/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-kube-api-access-9t6v6\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:52.573458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:52.573411 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fe195c-3ab8-4197-ae4d-5c3ca2452117-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:53.217219 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.217117 4227 generic.go:356] "Generic (PLEG): container finished" podID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerID="53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675" exitCode=0 Nov 03 21:17:53.217219 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.217199 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" Nov 03 21:17:53.217712 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.217258 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" event={"ID":"e5fe195c-3ab8-4197-ae4d-5c3ca2452117","Type":"ContainerDied","Data":"53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675"} Nov 03 21:17:53.217712 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.217299 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c" event={"ID":"e5fe195c-3ab8-4197-ae4d-5c3ca2452117","Type":"ContainerDied","Data":"76e562077fb715da4e4298ecdac8b32e80a5794c38efe4146128e5745e931446"} Nov 03 21:17:53.217712 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.217317 4227 scope.go:122] "RemoveContainer" containerID="53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675" Nov 03 21:17:53.218658 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.218631 4227 generic.go:356] "Generic (PLEG): container finished" podID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerID="44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67" exitCode=0 Nov 03 21:17:53.218772 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.218677 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" Nov 03 21:17:53.218772 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.218714 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" event={"ID":"f781fc90-550e-4649-b618-a1375b3cfbb8","Type":"ContainerDied","Data":"44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67"} Nov 03 21:17:53.218772 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.218747 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492" event={"ID":"f781fc90-550e-4649-b618-a1375b3cfbb8","Type":"ContainerDied","Data":"904b94dfe5326c194ec6184766b364ee87738d978ffb15782d0b77d03270447a"} Nov 03 21:17:53.220219 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.220189 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" event={"ID":"0352c748-5879-44cc-a69a-8fa2f1eedffa","Type":"ContainerDied","Data":"40f1888e1c933832c1d160e586a2d941058f919a3ea164f254df7aa8bebba870"} Nov 03 21:17:53.220298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.220242 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l" Nov 03 21:17:53.226116 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.226098 4227 scope.go:122] "RemoveContainer" containerID="53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675" Nov 03 21:17:53.226396 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:53.226377 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675\": container with ID starting with 53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675 not found: ID does not exist" containerID="53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675" Nov 03 21:17:53.226464 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.226405 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675"} err="failed to get container status \"53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675\": rpc error: code = NotFound desc = could not find container \"53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675\": container with ID starting with 53fcc25ea74916bfad1815ca02294515abc894698256e15a026a657b1e001675 not found: ID does not exist" Nov 03 21:17:53.226464 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.226420 4227 scope.go:122] "RemoveContainer" containerID="44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67" Nov 03 21:17:53.233711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.233689 4227 scope.go:122] "RemoveContainer" containerID="44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67" Nov 03 21:17:53.233943 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:53.233925 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67\": container with ID starting with 44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67 not found: ID does not exist" containerID="44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67" Nov 03 21:17:53.233985 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.233950 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67"} err="failed to get container status \"44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67\": rpc error: code = NotFound desc = could not find container \"44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67\": container with ID starting with 44c47eaa7e197157c086bf986bbec716791c9c97cd6c348369b5b3602297ed67 not found: ID does not exist" Nov 03 21:17:53.233985 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.233966 4227 scope.go:122] "RemoveContainer" containerID="e34495afed11661bd43e64b11f49ef924462471c97622c6c10bafba9ba4d9538" Nov 03 21:17:53.369638 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.369615 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c"] Nov 03 21:17:53.416056 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.416029 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/simulator-w3-fe2825-kserve-6d49744d7b-qtc4c"] Nov 03 21:17:53.472138 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.472070 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492"] Nov 03 21:17:53.474362 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.474339 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/premium-w3-fe2825-kserve-66b8989765-tm492"] Nov 03 21:17:53.575588 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.575561 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l"] Nov 03 21:17:53.581374 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:53.581350 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fe2825/distinct-2-w3-fe2825-kserve-56b696c8f5-x8c6l"] Nov 03 21:17:54.898669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:54.898637 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" path="/var/lib/kubelet/pods/0352c748-5879-44cc-a69a-8fa2f1eedffa/volumes" Nov 03 21:17:54.899036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:54.899021 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" path="/var/lib/kubelet/pods/e5fe195c-3ab8-4197-ae4d-5c3ca2452117/volumes" Nov 03 21:17:54.899410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:17:54.899397 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" path="/var/lib/kubelet/pods/f781fc90-550e-4649-b618-a1375b3cfbb8/volumes" Nov 03 21:17:56.083913 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:17:56.083877 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:18:02.757223 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.757186 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt"] Nov 03 21:18:02.757676 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.757476 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" containerID="cri-o://c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d" gracePeriod=30 Nov 03 21:18:02.802791 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.802757 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw"] Nov 03 21:18:02.803018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.802981 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" containerID="cri-o://b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d" gracePeriod=30 Nov 03 21:18:02.804070 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.804048 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm"] Nov 03 21:18:02.804239 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.804221 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" containerID="cri-o://25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e" gracePeriod=30 Nov 03 21:18:02.849875 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.849839 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq"] Nov 03 21:18:02.850135 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:02.850057 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" containerID="cri-o://836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3" gracePeriod=30 Nov 03 21:18:04.227457 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:04.227419 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-7rqsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:18:04.227836 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:04.227476 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:18:07.285255 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:07.285225 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz"] Nov 03 21:18:07.285732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:07.285465 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" containerID="cri-o://74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789" gracePeriod=30 Nov 03 21:18:07.882247 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:07.882217 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jsdpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:18:07.882359 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:07.882266 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:18:07.923441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:07.923407 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mlsmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:18:07.923542 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:07.923465 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:18:08.055108 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.055077 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 21:18:08.115602 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115572 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d396c328-e385-41d3-a1a0-ab54183aff2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d396c328-e385-41d3-a1a0-ab54183aff2b-istiod-ca-cert\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.115788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115622 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-podinfo\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.115788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115690 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-kube-api-access-dnchv\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-kube-api-access-dnchv\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.115788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115714 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-envoy\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.115788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115738 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-socket\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.115788 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115760 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-certs\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.116111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115796 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-credential-socket\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.116111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115817 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-data\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.116111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.115846 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-token\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-token\") pod \"d396c328-e385-41d3-a1a0-ab54183aff2b\" (UID: \"d396c328-e385-41d3-a1a0-ab54183aff2b\") " Nov 03 21:18:08.116746 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.116691 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-certs" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.117362 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.117142 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-socket" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.118394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.118366 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-data" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.118595 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.118566 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d396c328-e385-41d3-a1a0-ab54183aff2b-istiod-ca-cert" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:08.118595 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.118578 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-credential-socket" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.119923 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.119896 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-token" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.119923 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.119894 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-envoy" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.120226 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.120207 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-kube-api-access-dnchv" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "kube-api-access-dnchv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.121464 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.121437 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-podinfo" pod "d396c328-e385-41d3-a1a0-ab54183aff2b" (UID: "d396c328-e385-41d3-a1a0-ab54183aff2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:18:08.135380 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.135354 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:18:08.144389 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.144369 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 21:18:08.158152 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.158132 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:18:08.216633 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216600 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-kube-api-access-zkfsw\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-kube-api-access-zkfsw\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.216910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216695 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-kube-api-access-r4jsf\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-kube-api-access-r4jsf\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.216910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216749 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-certs\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.216910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216787 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-data\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.216910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216806 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-credential-socket\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.216910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216842 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-credential-socket\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.216910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216875 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-podinfo\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.216910 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216905 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-socket\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216941 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-envoy\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.216996 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-kube-api-access-2f4lz\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-kube-api-access-2f4lz\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217037 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-data\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217069 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-podinfo\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217079 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-certs" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217065 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-data" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217103 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7948db-b805-48a7-98b2-c3c27c227963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7948db-b805-48a7-98b2-c3c27c227963-istiod-ca-cert\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217132 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-certs\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217203 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-token\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-token\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217257 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b7948db-b805-48a7-98b2-c3c27c227963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7948db-b805-48a7-98b2-c3c27c227963-istio-podinfo\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217283 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-envoy\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217311 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca845828-d40b-4095-93d9-c5f7114bbdff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca845828-d40b-4095-93d9-c5f7114bbdff-istiod-ca-cert\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.217356 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217355 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-data\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217384 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-token\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-token\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217413 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-socket\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217448 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-credential-socket\") pod \"ca845828-d40b-4095-93d9-c5f7114bbdff\" (UID: \"ca845828-d40b-4095-93d9-c5f7114bbdff\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217482 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-socket\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217520 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istiod-ca-cert\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217566 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-certs\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217592 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-istio-token\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-istio-token\") pod \"5b7948db-b805-48a7-98b2-c3c27c227963\" (UID: \"5b7948db-b805-48a7-98b2-c3c27c227963\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217629 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-envoy\") pod \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\" (UID: \"c4b4c192-f4f6-4491-8853-00bc7b865b5d\") " Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217967 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217988 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218002 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218014 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218025 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218036 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218048 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218060 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d396c328-e385-41d3-a1a0-ab54183aff2b-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218072 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218086 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnchv\" (UniqueName: \"kubernetes.io/projected/d396c328-e385-41d3-a1a0-ab54183aff2b-kube-api-access-dnchv\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.218120 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.218100 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d396c328-e385-41d3-a1a0-ab54183aff2b-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.221311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217128 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-credential-socket" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.221311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217302 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-data" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.221311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217394 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-credential-socket" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.221311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.217491 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-socket" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.221311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.220667 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-certs" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.221311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.220682 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7948db-b805-48a7-98b2-c3c27c227963-istiod-ca-cert" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:08.221311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.220961 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-credential-socket" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.222001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.221325 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca845828-d40b-4095-93d9-c5f7114bbdff-istiod-ca-cert" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:08.222001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.221463 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-kube-api-access-zkfsw" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "kube-api-access-zkfsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.222001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.221609 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-socket" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.222001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.221639 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-certs" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.222001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.221663 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-socket" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.222001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.221675 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istiod-ca-cert" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:08.222001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.221808 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-data" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.222794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.222062 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b7948db-b805-48a7-98b2-c3c27c227963-istio-podinfo" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:18:08.222794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.222217 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-kube-api-access-2f4lz" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "kube-api-access-2f4lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.222794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.222538 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-podinfo" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:18:08.222794 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.222637 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-envoy" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.223090 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.223050 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-token" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.223324 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.223300 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-envoy" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.224053 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.224016 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-kube-api-access-r4jsf" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "kube-api-access-r4jsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.224558 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.224456 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-token" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.225080 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.225047 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-envoy" pod "ca845828-d40b-4095-93d9-c5f7114bbdff" (UID: "ca845828-d40b-4095-93d9-c5f7114bbdff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:08.225207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.225185 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-istio-token" pod "5b7948db-b805-48a7-98b2-c3c27c227963" (UID: "5b7948db-b805-48a7-98b2-c3c27c227963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:08.225487 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.225470 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-podinfo" pod "c4b4c192-f4f6-4491-8853-00bc7b865b5d" (UID: "c4b4c192-f4f6-4491-8853-00bc7b865b5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:18:08.293422 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.293339 4227 generic.go:356] "Generic (PLEG): container finished" podID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerID="b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d" exitCode=0 Nov 03 21:18:08.293422 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.293387 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" Nov 03 21:18:08.293944 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.293413 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" event={"ID":"c4b4c192-f4f6-4491-8853-00bc7b865b5d","Type":"ContainerDied","Data":"b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d"} Nov 03 21:18:08.293944 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.293502 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw" event={"ID":"c4b4c192-f4f6-4491-8853-00bc7b865b5d","Type":"ContainerDied","Data":"5a42e35a8bfb891181c0d0f04b50d77b8389205eda033132deec1ff0885f357d"} Nov 03 21:18:08.293944 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.293531 4227 scope.go:122] "RemoveContainer" containerID="b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d" Nov 03 21:18:08.297513 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.297489 4227 generic.go:356] "Generic (PLEG): container finished" podID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerID="c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d" exitCode=0 Nov 03 21:18:08.297637 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.297558 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" Nov 03 21:18:08.297637 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.297579 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" event={"ID":"d396c328-e385-41d3-a1a0-ab54183aff2b","Type":"ContainerDied","Data":"c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d"} Nov 03 21:18:08.297637 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.297632 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt" event={"ID":"d396c328-e385-41d3-a1a0-ab54183aff2b","Type":"ContainerDied","Data":"435307ac5fc3ec8c9c90c410a80784b16e0d641d3da856b77323039d66db07ee"} Nov 03 21:18:08.299104 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.299079 4227 generic.go:356] "Generic (PLEG): container finished" podID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerID="836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3" exitCode=0 Nov 03 21:18:08.299277 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.299232 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" Nov 03 21:18:08.299469 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.299444 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" event={"ID":"ca845828-d40b-4095-93d9-c5f7114bbdff","Type":"ContainerDied","Data":"836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3"} Nov 03 21:18:08.299469 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.299492 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq" event={"ID":"ca845828-d40b-4095-93d9-c5f7114bbdff","Type":"ContainerDied","Data":"22d455c909a7efd3e3bc4a44dbbae6b3d15c9d554705571319d73aee6fe52170"} Nov 03 21:18:08.304037 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.303810 4227 generic.go:356] "Generic (PLEG): container finished" podID="5b7948db-b805-48a7-98b2-c3c27c227963" containerID="25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e" exitCode=0 Nov 03 21:18:08.304037 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.303839 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" event={"ID":"5b7948db-b805-48a7-98b2-c3c27c227963","Type":"ContainerDied","Data":"25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e"} Nov 03 21:18:08.304037 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.303887 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" event={"ID":"5b7948db-b805-48a7-98b2-c3c27c227963","Type":"ContainerDied","Data":"97f8acce098e4b92e02ccbb26e4b2c667d4e6a4b3b6bde461bac2ea610918330"} Nov 03 21:18:08.304037 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.303971 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm" Nov 03 21:18:08.306939 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.306918 4227 scope.go:122] "RemoveContainer" containerID="b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d" Nov 03 21:18:08.307254 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:18:08.307230 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d\": container with ID starting with b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d not found: ID does not exist" containerID="b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d" Nov 03 21:18:08.307350 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.307267 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d"} err="failed to get container status \"b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d\": rpc error: code = NotFound desc = could not find container \"b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d\": container with ID starting with b63d32b3203f53bbff43a29186e6cc3ca140ec8744ea5e87bd387be83fa7d75d not found: ID does not exist" Nov 03 21:18:08.307350 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.307290 4227 scope.go:122] "RemoveContainer" containerID="c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d" Nov 03 21:18:08.316262 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.316244 4227 scope.go:122] "RemoveContainer" containerID="c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d" Nov 03 21:18:08.316505 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:18:08.316483 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d\": container with ID starting with c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d not found: ID does not exist" containerID="c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d" Nov 03 21:18:08.316580 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.316515 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d"} err="failed to get container status \"c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d\": rpc error: code = NotFound desc = could not find container \"c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d\": container with ID starting with c2a4baaa55da1672f731820bd3f772cac9c26756f68ba20dea4ca10183e78e7d not found: ID does not exist" Nov 03 21:18:08.316580 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.316535 4227 scope.go:122] "RemoveContainer" containerID="836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3" Nov 03 21:18:08.318801 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318777 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.318877 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318808 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfsw\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-kube-api-access-zkfsw\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.318877 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318824 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4jsf\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-kube-api-access-r4jsf\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.318877 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318839 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.318877 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318852 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.318877 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318865 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318880 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318895 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318908 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f4lz\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-kube-api-access-2f4lz\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318923 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318936 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318949 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7948db-b805-48a7-98b2-c3c27c227963-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318964 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318977 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.318991 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7948db-b805-48a7-98b2-c3c27c227963-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319004 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319017 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca845828-d40b-4095-93d9-c5f7114bbdff-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319036 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319031 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319045 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca845828-d40b-4095-93d9-c5f7114bbdff-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319058 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319071 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca845828-d40b-4095-93d9-c5f7114bbdff-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319084 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7948db-b805-48a7-98b2-c3c27c227963-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319099 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b4c192-f4f6-4491-8853-00bc7b865b5d-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319114 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b4c192-f4f6-4491-8853-00bc7b865b5d-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.319425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.319127 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7948db-b805-48a7-98b2-c3c27c227963-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:08.324674 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.324657 4227 scope.go:122] "RemoveContainer" containerID="836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3" Nov 03 21:18:08.324908 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:18:08.324890 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3\": container with ID starting with 836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3 not found: ID does not exist" containerID="836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3" Nov 03 21:18:08.324957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.324913 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3"} err="failed to get container status \"836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3\": rpc error: code = NotFound desc = could not find container \"836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3\": container with ID starting with 836d0fbcb531d077e9bb0f508ca0c78436b6de0ba7dd8b9888246addc01105b3 not found: ID does not exist" Nov 03 21:18:08.324957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.324927 4227 scope.go:122] "RemoveContainer" containerID="25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e" Nov 03 21:18:08.332221 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.332204 4227 scope.go:122] "RemoveContainer" containerID="25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e" Nov 03 21:18:08.332475 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:18:08.332458 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e\": container with ID starting with 25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e not found: ID does not exist" containerID="25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e" Nov 03 21:18:08.332524 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.332480 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e"} err="failed to get container status \"25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e\": rpc error: code = NotFound desc = could not find container \"25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e\": container with ID starting with 25fc08df132c356ce88398b85856f5e73414acf59784641a8237b4a808911b1e not found: ID does not exist" Nov 03 21:18:08.599065 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.598992 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt"] Nov 03 21:18:08.602334 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.602308 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-lpszt"] Nov 03 21:18:08.704329 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.704298 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq"] Nov 03 21:18:08.707749 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.707721 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-7rqsq"] Nov 03 21:18:08.810799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.810769 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw"] Nov 03 21:18:08.816344 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.816320 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jsdpw"] Nov 03 21:18:08.899025 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.899000 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" path="/var/lib/kubelet/pods/c4b4c192-f4f6-4491-8853-00bc7b865b5d/volumes" Nov 03 21:18:08.899467 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.899452 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" path="/var/lib/kubelet/pods/ca845828-d40b-4095-93d9-c5f7114bbdff/volumes" Nov 03 21:18:08.899832 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.899820 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" path="/var/lib/kubelet/pods/d396c328-e385-41d3-a1a0-ab54183aff2b/volumes" Nov 03 21:18:08.917890 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.917864 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm"] Nov 03 21:18:08.921500 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:08.921479 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mlsmm"] Nov 03 21:18:10.897651 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:10.897617 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" path="/var/lib/kubelet/pods/5b7948db-b805-48a7-98b2-c3c27c227963/volumes" Nov 03 21:18:12.524393 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.524362 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 21:18:12.656399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656375 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-kube-api-access-c6pxk\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-kube-api-access-c6pxk\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656413 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-socket\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656440 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-certs\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656464 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-credential-socket\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656498 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-token\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-token\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656539 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656533 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00e00e0-ff18-434c-8531-55eccd69c74f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00e00e0-ff18-434c-8531-55eccd69c74f-istiod-ca-cert\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656814 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656562 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-data\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656814 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656611 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-podinfo\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656814 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656662 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-envoy\") pod \"f00e00e0-ff18-434c-8531-55eccd69c74f\" (UID: \"f00e00e0-ff18-434c-8531-55eccd69c74f\") " Nov 03 21:18:12.656814 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656688 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-socket" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:12.656814 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656711 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-certs" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:12.657083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656894 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-data" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:12.657083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656909 4227 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.657083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.656926 4227 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-workload-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.657083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.657017 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-credential-socket" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:12.657083 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.657064 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00e00e0-ff18-434c-8531-55eccd69c74f-istiod-ca-cert" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:12.658663 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.658642 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-token" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:12.658741 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.658681 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-envoy" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:12.658741 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.658726 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-podinfo" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:18:12.658817 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.658791 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-kube-api-access-c6pxk" pod "f00e00e0-ff18-434c-8531-55eccd69c74f" (UID: "f00e00e0-ff18-434c-8531-55eccd69c74f"). InnerVolumeSpecName "kube-api-access-c6pxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:12.757468 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.757441 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6pxk\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-kube-api-access-c6pxk\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.757468 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.757464 4227 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-credential-socket\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.757609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.757473 4227 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-token\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.757609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.757484 4227 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00e00e0-ff18-434c-8531-55eccd69c74f-istiod-ca-cert\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.757609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.757495 4227 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-data\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.757609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.757504 4227 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-podinfo\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:12.757609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:12.757512 4227 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00e00e0-ff18-434c-8531-55eccd69c74f-istio-envoy\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:13.332449 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.332357 4227 generic.go:356] "Generic (PLEG): container finished" podID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerID="74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789" exitCode=0 Nov 03 21:18:13.332449 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.332397 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" event={"ID":"f00e00e0-ff18-434c-8531-55eccd69c74f","Type":"ContainerDied","Data":"74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789"} Nov 03 21:18:13.332679 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.332443 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" Nov 03 21:18:13.332679 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.332458 4227 scope.go:122] "RemoveContainer" containerID="74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789" Nov 03 21:18:13.332679 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.332448 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz" event={"ID":"f00e00e0-ff18-434c-8531-55eccd69c74f","Type":"ContainerDied","Data":"308019c84c4a426a80a890b0edff8986a3b5665faa1debfcefdbad3ff9483182"} Nov 03 21:18:13.343820 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.343798 4227 scope.go:122] "RemoveContainer" containerID="74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789" Nov 03 21:18:13.344170 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:18:13.344130 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789\": container with ID starting with 74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789 not found: ID does not exist" containerID="74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789" Nov 03 21:18:13.344261 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.344191 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789"} err="failed to get container status \"74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789\": rpc error: code = NotFound desc = could not find container \"74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789\": container with ID starting with 74afe8b982514049e6cc51632d7424bd1629878858c7da35f5c0a12a2b6bf789 not found: ID does not exist" Nov 03 21:18:13.531984 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.531955 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz"] Nov 03 21:18:13.578108 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:13.578077 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-qzvjz"] Nov 03 21:18:14.900993 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:14.900958 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" path="/var/lib/kubelet/pods/f00e00e0-ff18-434c-8531-55eccd69c74f/volumes" Nov 03 21:18:26.080392 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:18:26.080350 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:18:36.921258 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921224 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g"] Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921515 4227 cpu_manager.go:414] "containerMap: removing container" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921526 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921536 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921542 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921550 4227 cpu_manager.go:414] "containerMap: removing container" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921555 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921563 4227 cpu_manager.go:414] "containerMap: removing container" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921567 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921580 4227 cpu_manager.go:414] "containerMap: removing container" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921585 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921592 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921597 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921607 4227 cpu_manager.go:414] "containerMap: removing container" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921612 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921618 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" Nov 03 21:18:36.921619 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921622 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921631 4227 cpu_manager.go:414] "containerMap: removing container" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921635 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921645 4227 cpu_manager.go:414] "containerMap: removing container" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921649 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921658 4227 cpu_manager.go:414] "containerMap: removing container" podUID="6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" containerName="revoke-keys" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921662 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" containerName="revoke-keys" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921710 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7948db-b805-48a7-98b2-c3c27c227963" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921717 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6278553-4b48-4f8c-bbc3-d3217718265c" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921723 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="385d99b9-2716-4795-9c7a-f71259e86d6f" containerName="cleanup" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921728 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="d396c328-e385-41d3-a1a0-ab54183aff2b" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921735 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c84ab14-c774-41c4-852a-4eb8698c83ea" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921741 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="0352c748-5879-44cc-a69a-8fa2f1eedffa" containerName="main" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921746 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b4c192-f4f6-4491-8853-00bc7b865b5d" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921752 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fe195c-3ab8-4197-ae4d-5c3ca2452117" containerName="main" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921757 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f781fc90-550e-4649-b618-a1375b3cfbb8" containerName="main" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921762 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00e00e0-ff18-434c-8531-55eccd69c74f" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921768 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca845828-d40b-4095-93d9-c5f7114bbdff" containerName="istio-proxy" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921772 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d154031-55f8-4ac9-93d4-8ea3d7d3a958" containerName="cleanup" Nov 03 21:18:36.922117 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.921778 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" containerName="revoke-keys" Nov 03 21:18:36.924426 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.924404 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:36.989438 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.989411 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:18:36.989571 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.989435 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:18:36.989571 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.989437 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:18:36.990799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:36.990778 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g"] Nov 03 21:18:37.022217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.022198 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:18:37.043946 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.043921 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.044058 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.043951 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smgz8\" (UniqueName: \"kubernetes.io/projected/75b1d86f-0f54-4b2b-9631-68080548aefd-kube-api-access-smgz8\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.044127 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.044094 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.144828 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.144803 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.144925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.144843 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.144925 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.144860 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smgz8\" (UniqueName: \"kubernetes.io/projected/75b1d86f-0f54-4b2b-9631-68080548aefd-kube-api-access-smgz8\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.145421 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.145402 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.147040 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.147023 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.198072 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.198015 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smgz8\" (UniqueName: \"kubernetes.io/projected/75b1d86f-0f54-4b2b-9631-68080548aefd-kube-api-access-smgz8\") pod \"maas-api-revoke-sub-85237e45-f4w2g\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.236880 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.236854 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:37.600312 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:37.600222 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g"] Nov 03 21:18:37.608558 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:18:37.608524 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b1d86f_0f54_4b2b_9631_68080548aefd.slice/crio-be2e5c3714a5d926b975ef504d5c4b7fbec9c485798ecf21284357af938e0b19 WatchSource:0}: Error finding container be2e5c3714a5d926b975ef504d5c4b7fbec9c485798ecf21284357af938e0b19: Status 404 returned error can't find the container with id be2e5c3714a5d926b975ef504d5c4b7fbec9c485798ecf21284357af938e0b19 Nov 03 21:18:38.438452 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:38.438421 4227 generic.go:356] "Generic (PLEG): container finished" podID="75b1d86f-0f54-4b2b-9631-68080548aefd" containerID="70893a472454403571239792f37eff70aedf2365d5f278ce1842dacd702971b2" exitCode=0 Nov 03 21:18:38.438769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:38.438498 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" event={"ID":"75b1d86f-0f54-4b2b-9631-68080548aefd","Type":"ContainerDied","Data":"70893a472454403571239792f37eff70aedf2365d5f278ce1842dacd702971b2"} Nov 03 21:18:38.438769 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:38.438556 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" event={"ID":"75b1d86f-0f54-4b2b-9631-68080548aefd","Type":"ContainerStarted","Data":"be2e5c3714a5d926b975ef504d5c4b7fbec9c485798ecf21284357af938e0b19"} Nov 03 21:18:39.563772 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.563747 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:39.665027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.665002 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b1d86f-0f54-4b2b-9631-68080548aefd-kube-api-access-smgz8\" (UniqueName: \"kubernetes.io/projected/75b1d86f-0f54-4b2b-9631-68080548aefd-kube-api-access-smgz8\") pod \"75b1d86f-0f54-4b2b-9631-68080548aefd\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " Nov 03 21:18:39.665153 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.665044 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-xks-ca\") pod \"75b1d86f-0f54-4b2b-9631-68080548aefd\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " Nov 03 21:18:39.665153 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.665115 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-openshift-ca\") pod \"75b1d86f-0f54-4b2b-9631-68080548aefd\" (UID: \"75b1d86f-0f54-4b2b-9631-68080548aefd\") " Nov 03 21:18:39.665460 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.665439 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-openshift-ca" pod "75b1d86f-0f54-4b2b-9631-68080548aefd" (UID: "75b1d86f-0f54-4b2b-9631-68080548aefd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:39.667067 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.667048 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b1d86f-0f54-4b2b-9631-68080548aefd-kube-api-access-smgz8" pod "75b1d86f-0f54-4b2b-9631-68080548aefd" (UID: "75b1d86f-0f54-4b2b-9631-68080548aefd"). InnerVolumeSpecName "kube-api-access-smgz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:39.667182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.667132 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-xks-ca" pod "75b1d86f-0f54-4b2b-9631-68080548aefd" (UID: "75b1d86f-0f54-4b2b-9631-68080548aefd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:39.766399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.766348 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smgz8\" (UniqueName: \"kubernetes.io/projected/75b1d86f-0f54-4b2b-9631-68080548aefd-kube-api-access-smgz8\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:39.766399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.766369 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-xks-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:39.766399 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:39.766379 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b1d86f-0f54-4b2b-9631-68080548aefd-maas-api-openshift-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:40.447577 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:40.447542 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" event={"ID":"75b1d86f-0f54-4b2b-9631-68080548aefd","Type":"ContainerDied","Data":"be2e5c3714a5d926b975ef504d5c4b7fbec9c485798ecf21284357af938e0b19"} Nov 03 21:18:40.447577 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:40.447572 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g" Nov 03 21:18:40.447790 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:40.447593 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2e5c3714a5d926b975ef504d5c4b7fbec9c485798ecf21284357af938e0b19" Nov 03 21:18:48.103921 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.103879 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r"] Nov 03 21:18:48.104437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.104397 4227 cpu_manager.go:414] "containerMap: removing container" podUID="75b1d86f-0f54-4b2b-9631-68080548aefd" containerName="revoke-keys" Nov 03 21:18:48.104437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.104426 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b1d86f-0f54-4b2b-9631-68080548aefd" containerName="revoke-keys" Nov 03 21:18:48.104813 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.104530 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b1d86f-0f54-4b2b-9631-68080548aefd" containerName="revoke-keys" Nov 03 21:18:48.107697 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.107675 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.128895 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.128872 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129008 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.128904 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfxt\" (UniqueName: \"kubernetes.io/projected/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-kube-api-access-grfxt\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129008 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.128933 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129133 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.129053 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129133 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.129090 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129133 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.129125 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129326 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.129153 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129326 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.129247 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.129326 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.129299 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.150928 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.150902 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z"] Nov 03 21:18:48.159332 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.159262 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh"] Nov 03 21:18:48.159441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.159397 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.162810 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.162795 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.195827 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.195807 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spnvt\"" Nov 03 21:18:48.202720 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.202695 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r"] Nov 03 21:18:48.230634 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230607 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.230738 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230644 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7d5\" (UniqueName: \"kubernetes.io/projected/566679d2-822c-4dfc-b266-459ba037cdba-kube-api-access-jv7d5\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.230738 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230662 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.230738 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230679 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.230908 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230736 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.230908 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230779 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.230908 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230874 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zvb\" (UniqueName: \"kubernetes.io/projected/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-kube-api-access-22zvb\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.230908 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230904 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.231082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.230990 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.231082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231042 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231061 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231292 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231080 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231292 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231120 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.231292 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231183 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.231292 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231226 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231292 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231270 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231488 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231313 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231488 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231376 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566679d2-822c-4dfc-b266-459ba037cdba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.231488 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231407 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231488 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231424 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231488 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231434 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231514 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231536 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231546 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231577 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231659 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566679d2-822c-4dfc-b266-459ba037cdba-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.231796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231682 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231699 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566679d2-822c-4dfc-b266-459ba037cdba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.231796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231744 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.231796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231746 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.231796 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231784 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.232008 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.231816 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfxt\" (UniqueName: \"kubernetes.io/projected/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-kube-api-access-grfxt\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.233271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.233250 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.233611 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.233594 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.249894 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.249871 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z"] Nov 03 21:18:48.253897 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.253875 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh"] Nov 03 21:18:48.283741 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.283718 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.284559 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.284540 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfxt\" (UniqueName: \"kubernetes.io/projected/5f02aedc-87ae-4383-a34c-40f3cbe4eef9-kube-api-access-grfxt\") pod \"maas-default-gateway-openshift-default-696f47c6c-rh44r\" (UID: \"5f02aedc-87ae-4383-a34c-40f3cbe4eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.332198 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332150 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332214 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332235 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332257 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566679d2-822c-4dfc-b266-459ba037cdba-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.332323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332282 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332323 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332307 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566679d2-822c-4dfc-b266-459ba037cdba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332331 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332390 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7d5\" (UniqueName: \"kubernetes.io/projected/566679d2-822c-4dfc-b266-459ba037cdba-kube-api-access-jv7d5\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332434 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332466 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zvb\" (UniqueName: \"kubernetes.io/projected/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-kube-api-access-22zvb\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332497 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332525 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332556 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332573 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332585 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.332627 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332628 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.333150 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332631 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.333150 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332680 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.333150 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332709 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.333150 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.332746 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566679d2-822c-4dfc-b266-459ba037cdba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.333150 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.333004 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.333150 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.333050 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.333609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.333586 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.333740 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.333715 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.333809 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.333737 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.333872 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.333826 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.334074 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.334055 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.334485 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.334461 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566679d2-822c-4dfc-b266-459ba037cdba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.335188 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.335123 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.335309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.335250 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.336104 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.336083 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566679d2-822c-4dfc-b266-459ba037cdba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.336658 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.336637 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566679d2-822c-4dfc-b266-459ba037cdba-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.385874 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.385803 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zvb\" (UniqueName: \"kubernetes.io/projected/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-kube-api-access-22zvb\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.386234 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.386215 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566679d2-822c-4dfc-b266-459ba037cdba-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.386560 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.386543 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f780b9d-4c02-4e0f-9cec-9429a0ee4619-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-ljp6z\" (UID: \"1f780b9d-4c02-4e0f-9cec-9429a0ee4619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.386631 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.386587 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7d5\" (UniqueName: \"kubernetes.io/projected/566679d2-822c-4dfc-b266-459ba037cdba-kube-api-access-jv7d5\") pod \"maas-default-gateway-openshift-default-696f47c6c-skskh\" (UID: \"566679d2-822c-4dfc-b266-459ba037cdba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.419139 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.419117 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:48.472436 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.472399 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:48.480395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.479431 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:48.787755 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.787718 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r"] Nov 03 21:18:48.794515 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:18:48.794469 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f02aedc_87ae_4383_a34c_40f3cbe4eef9.slice/crio-8e3b5a134ddbc5d30960e8957bc505f3874b039aa2dd2844fa5dc8ed22b3567b WatchSource:0}: Error finding container 8e3b5a134ddbc5d30960e8957bc505f3874b039aa2dd2844fa5dc8ed22b3567b: Status 404 returned error can't find the container with id 8e3b5a134ddbc5d30960e8957bc505f3874b039aa2dd2844fa5dc8ed22b3567b Nov 03 21:18:48.799518 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.799487 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.799629 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.799551 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.799629 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.799582 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.903313 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.903276 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh"] Nov 03 21:18:48.911263 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:18:48.911234 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566679d2_822c_4dfc_b266_459ba037cdba.slice/crio-8b28e7037e3ac280ae135ebae1cd54a17d7612306b3ff726b8a104cf04ad75a6 WatchSource:0}: Error finding container 8b28e7037e3ac280ae135ebae1cd54a17d7612306b3ff726b8a104cf04ad75a6: Status 404 returned error can't find the container with id 8b28e7037e3ac280ae135ebae1cd54a17d7612306b3ff726b8a104cf04ad75a6 Nov 03 21:18:48.913963 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.913931 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.914058 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.914014 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.914097 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.914058 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.944517 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.944491 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z"] Nov 03 21:18:48.966261 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:18:48.966234 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f780b9d_4c02_4e0f_9cec_9429a0ee4619.slice/crio-939facd020d52a5b3416edc8935a0fb7d5ed2f5b41209d7b4a4969bcc985e036 WatchSource:0}: Error finding container 939facd020d52a5b3416edc8935a0fb7d5ed2f5b41209d7b4a4969bcc985e036: Status 404 returned error can't find the container with id 939facd020d52a5b3416edc8935a0fb7d5ed2f5b41209d7b4a4969bcc985e036 Nov 03 21:18:48.968571 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.968531 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.968685 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.968606 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:48.968685 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:48.968647 4227 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:18:49.496190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.495487 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" event={"ID":"5f02aedc-87ae-4383-a34c-40f3cbe4eef9","Type":"ContainerStarted","Data":"8bcb183defe26722f5501e83344db3c0805b9e87a0fe072a61c51bd5aa1daa06"} Nov 03 21:18:49.496190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.495557 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" event={"ID":"5f02aedc-87ae-4383-a34c-40f3cbe4eef9","Type":"ContainerStarted","Data":"8e3b5a134ddbc5d30960e8957bc505f3874b039aa2dd2844fa5dc8ed22b3567b"} Nov 03 21:18:49.499448 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.498837 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" event={"ID":"1f780b9d-4c02-4e0f-9cec-9429a0ee4619","Type":"ContainerStarted","Data":"c0f44b9f287342e4fceb985f56feb6535206bf3cce659f65f88ec86d0fc18356"} Nov 03 21:18:49.499448 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.498892 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" event={"ID":"1f780b9d-4c02-4e0f-9cec-9429a0ee4619","Type":"ContainerStarted","Data":"939facd020d52a5b3416edc8935a0fb7d5ed2f5b41209d7b4a4969bcc985e036"} Nov 03 21:18:49.501711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.501206 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" event={"ID":"566679d2-822c-4dfc-b266-459ba037cdba","Type":"ContainerStarted","Data":"f3717881d902fe3bbff4435b642ee18a15c82056c6d99c0f4be1cd8610d2e499"} Nov 03 21:18:49.501711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.501253 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" event={"ID":"566679d2-822c-4dfc-b266-459ba037cdba","Type":"ContainerStarted","Data":"8b28e7037e3ac280ae135ebae1cd54a17d7612306b3ff726b8a104cf04ad75a6"} Nov 03 21:18:49.609185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.608956 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podStartSLOduration=1.608936863 podStartE2EDuration="1.608936863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:49.606711509 +0000 UTC m=+2996.843689384" watchObservedRunningTime="2026-11-03 21:18:49.608936863 +0000 UTC m=+2996.845914740" Nov 03 21:18:49.718183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.716317 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podStartSLOduration=1.7162964120000002 podStartE2EDuration="1.716296412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:49.715692058 +0000 UTC m=+2996.952669934" watchObservedRunningTime="2026-11-03 21:18:49.716296412 +0000 UTC m=+2996.953274288" Nov 03 21:18:49.830217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:49.830064 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podStartSLOduration=1.830042487 podStartE2EDuration="1.830042487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:49.827961253 +0000 UTC m=+2997.064939130" watchObservedRunningTime="2026-11-03 21:18:49.830042487 +0000 UTC m=+2997.067020362" Nov 03 21:18:50.421184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.420265 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:50.421184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.420991 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:18:50.421184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.421041 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:18:50.476183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.475485 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:50.489207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.488295 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-ljp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:18:50.489207 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.488360 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podUID="1f780b9d-4c02-4e0f-9cec-9429a0ee4619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 21:18:50.492201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.489871 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:50.492201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.489988 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:18:50.492201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:50.490022 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podUID="566679d2-822c-4dfc-b266-459ba037cdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 21:18:51.422517 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:51.422479 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:18:51.423038 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:51.422534 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:18:51.486184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:51.478038 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-ljp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:18:51.486184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:51.478098 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podUID="1f780b9d-4c02-4e0f-9cec-9429a0ee4619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 21:18:51.486184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:51.482455 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:18:51.486184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:51.482497 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podUID="566679d2-822c-4dfc-b266-459ba037cdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 21:18:52.419657 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:52.419611 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:18:52.419998 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:52.419954 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:18:52.477183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:52.476056 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-ljp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:18:52.477183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:52.476110 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podUID="1f780b9d-4c02-4e0f-9cec-9429a0ee4619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 21:18:52.486532 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:52.486502 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:18:52.486746 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:52.486721 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podUID="566679d2-822c-4dfc-b266-459ba037cdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 21:18:53.271187 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.267769 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:18:53.321795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.321767 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:18:53.420183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.419592 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:18:53.420183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.419657 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:18:53.476184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.475528 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-ljp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:18:53.476184 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.475588 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podUID="1f780b9d-4c02-4e0f-9cec-9429a0ee4619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 21:18:53.488186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.487938 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:18:53.488186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:53.487988 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podUID="566679d2-822c-4dfc-b266-459ba037cdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 21:18:54.420185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:54.419638 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:18:54.420185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:54.419713 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:18:54.477183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:54.477039 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-ljp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:18:54.477183 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:54.477096 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podUID="1f780b9d-4c02-4e0f-9cec-9429a0ee4619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 21:18:54.485182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:54.481454 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:18:54.485182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:54.481500 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podUID="566679d2-822c-4dfc-b266-459ba037cdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 21:18:55.419746 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:55.419705 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:18:55.420148 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:55.419777 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:18:55.473233 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:55.473198 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-ljp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:18:55.473379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:55.473264 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podUID="1f780b9d-4c02-4e0f-9cec-9429a0ee4619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 21:18:55.480508 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:55.480481 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:18:55.480609 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:55.480534 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podUID="566679d2-822c-4dfc-b266-459ba037cdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 21:18:56.086370 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:18:56.086325 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:18:56.420189 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:56.420132 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:18:56.420590 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:56.420218 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:18:56.473395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:56.473356 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-ljp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:18:56.473695 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:56.473656 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" podUID="1f780b9d-4c02-4e0f-9cec-9429a0ee4619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 21:18:56.480767 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:56.480735 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:18:56.480895 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:56.480786 4227 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" podUID="566679d2-822c-4dfc-b266-459ba037cdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 21:18:57.518559 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.518527 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:57.519368 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.519303 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:57.520343 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.520316 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" Nov 03 21:18:57.651237 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.651194 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:57.652829 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.652031 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:57.652829 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.652793 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skskh" Nov 03 21:18:57.711394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.711365 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:57.712138 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.712116 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:18:57.712722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:18:57.712702 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-ljp6z" Nov 03 21:19:26.081434 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:19:26.081397 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:19:56.079913 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:19:56.079875 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:20:26.080635 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:20:26.080598 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:20:56.080870 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:20:56.080832 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:21:26.080875 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:21:26.080837 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:21:56.082408 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:21:56.082372 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:22:26.081205 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:22:26.081150 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:22:48.044601 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:22:48.044563 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl"] Nov 03 21:22:48.098851 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:22:48.098818 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86eb7c8b-f4mkl"] Nov 03 21:22:48.898379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:22:48.898345 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0ba26d-e248-4f02-a7cf-58121f1b9d1d" path="/var/lib/kubelet/pods/6f0ba26d-e248-4f02-a7cf-58121f1b9d1d/volumes" Nov 03 21:22:56.082282 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:22:56.082239 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:23:26.080974 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:23:26.080932 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:23:40.049326 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:23:40.049242 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g"] Nov 03 21:23:40.095783 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:23:40.095748 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f4w2g"] Nov 03 21:23:40.898060 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:23:40.898028 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b1d86f-0f54-4b2b-9631-68080548aefd" path="/var/lib/kubelet/pods/75b1d86f-0f54-4b2b-9631-68080548aefd/volumes" Nov 03 21:23:53.341342 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:23:53.341310 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:23:53.352292 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:23:53.352270 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:23:54.066053 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:23:54.066022 4227 scope.go:122] "RemoveContainer" containerID="d8e9eafb146b92e008af0b625bcd51fde9c2e560b3a0a936fbef2d9e3c1d76ef" Nov 03 21:23:56.079959 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:23:56.079918 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:24:26.081313 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:24:26.081273 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:24:54.084202 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:24:54.084147 4227 scope.go:122] "RemoveContainer" containerID="70893a472454403571239792f37eff70aedf2365d5f278ce1842dacd702971b2" Nov 03 21:24:56.080955 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:24:56.080921 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:25:26.082067 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:25:26.082030 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:25:56.082119 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:25:56.082004 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:26:12.477553 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.477516 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc"] Nov 03 21:26:12.480757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.480724 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.576233 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.576202 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d9f39a\"/\"kube-root-ca.crt\"" Nov 03 21:26:12.576233 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.576203 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d9f39a\"/\"openshift-service-ca.crt\"" Nov 03 21:26:12.576521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.576320 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d9f39a\"/\"simulator-single-main-d9f39a-kserve-self-signed-certs\"" Nov 03 21:26:12.576521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.576493 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d9f39a\"/\"default-dockercfg-49wbb\"" Nov 03 21:26:12.577896 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.577871 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc"] Nov 03 21:26:12.578228 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.578209 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjm4c\" (UniqueName: \"kubernetes.io/projected/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-kube-api-access-gjm4c\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.578271 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.578239 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-home\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.578324 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.578266 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tls-certs\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.578384 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.578372 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tmp-dir\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.578420 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.578393 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-dshm\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.578468 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.578443 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-model-cache\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.679750 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.679716 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjm4c\" (UniqueName: \"kubernetes.io/projected/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-kube-api-access-gjm4c\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.679750 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.679752 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-home\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.680030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.679774 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tls-certs\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.680030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.679819 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tmp-dir\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.680030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.679844 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-dshm\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.680030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.679912 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-model-cache\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.680305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.680260 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tmp-dir\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.680305 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.680260 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-home\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.680412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.680370 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-model-cache\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.682108 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.682083 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-dshm\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.682386 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.682369 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tls-certs\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.733116 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.733040 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjm4c\" (UniqueName: \"kubernetes.io/projected/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-kube-api-access-gjm4c\") pod \"simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:12.809042 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:12.808999 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:13.181807 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:13.181771 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc"] Nov 03 21:26:13.188711 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:26:13.188684 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f2a9bae_7ad2_45c5_a293_6090633cb6c7.slice/crio-d7c872e29d5d38c2cb863d1ef1e0fa207e06314d39cccfa3981dbd269ecdcb9d WatchSource:0}: Error finding container d7c872e29d5d38c2cb863d1ef1e0fa207e06314d39cccfa3981dbd269ecdcb9d: Status 404 returned error can't find the container with id d7c872e29d5d38c2cb863d1ef1e0fa207e06314d39cccfa3981dbd269ecdcb9d Nov 03 21:26:13.703457 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:13.703424 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" event={"ID":"7f2a9bae-7ad2-45c5-a293-6090633cb6c7","Type":"ContainerStarted","Data":"a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747"} Nov 03 21:26:13.703457 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:13.703462 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" event={"ID":"7f2a9bae-7ad2-45c5-a293-6090633cb6c7","Type":"ContainerStarted","Data":"d7c872e29d5d38c2cb863d1ef1e0fa207e06314d39cccfa3981dbd269ecdcb9d"} Nov 03 21:26:13.809140 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:13.809112 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:13.857277 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:13.857225 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" podStartSLOduration=1.857212071 podStartE2EDuration="1.857212071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:13.809522211 +0000 UTC m=+3441.046500085" watchObservedRunningTime="2026-11-03 21:26:13.857212071 +0000 UTC m=+3441.094189923" Nov 03 21:26:22.809632 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:22.809594 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:22.821979 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:22.821950 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:23.755797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:26:23.755761 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:26:26.080811 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:26:26.080771 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:26:56.079594 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:26:56.079551 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:27:13.473219 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:13.473187 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc"] Nov 03 21:27:26.081412 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:27:26.081371 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:27:28.492862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.492778 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" podUID="7f2a9bae-7ad2-45c5-a293-6090633cb6c7" containerName="main" containerID="cri-o://a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747" gracePeriod=45 Nov 03 21:27:28.733330 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.733307 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:27:28.798338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798265 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-model-cache\") pod \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " Nov 03 21:27:28.798338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798296 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-dshm\") pod \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " Nov 03 21:27:28.798338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798320 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tls-certs\") pod \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " Nov 03 21:27:28.798624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798343 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tmp-dir\") pod \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " Nov 03 21:27:28.798624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798411 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-home\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-home\") pod \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " Nov 03 21:27:28.798624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798443 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-kube-api-access-gjm4c\" (UniqueName: \"kubernetes.io/projected/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-kube-api-access-gjm4c\") pod \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\" (UID: \"7f2a9bae-7ad2-45c5-a293-6090633cb6c7\") " Nov 03 21:27:28.798624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798560 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-model-cache" pod "7f2a9bae-7ad2-45c5-a293-6090633cb6c7" (UID: "7f2a9bae-7ad2-45c5-a293-6090633cb6c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:28.798833 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798674 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-home" pod "7f2a9bae-7ad2-45c5-a293-6090633cb6c7" (UID: "7f2a9bae-7ad2-45c5-a293-6090633cb6c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:28.798833 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798735 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tmp-dir" pod "7f2a9bae-7ad2-45c5-a293-6090633cb6c7" (UID: "7f2a9bae-7ad2-45c5-a293-6090633cb6c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:28.798833 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.798747 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:28.800456 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.800435 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-dshm" pod "7f2a9bae-7ad2-45c5-a293-6090633cb6c7" (UID: "7f2a9bae-7ad2-45c5-a293-6090633cb6c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:27:28.800558 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.800510 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tls-certs" pod "7f2a9bae-7ad2-45c5-a293-6090633cb6c7" (UID: "7f2a9bae-7ad2-45c5-a293-6090633cb6c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:28.800621 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.800599 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-kube-api-access-gjm4c" pod "7f2a9bae-7ad2-45c5-a293-6090633cb6c7" (UID: "7f2a9bae-7ad2-45c5-a293-6090633cb6c7"). InnerVolumeSpecName "kube-api-access-gjm4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:28.900054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.900016 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:28.900054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.900044 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:28.900054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.900054 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:28.900054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.900062 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:28.900342 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:28.900242 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjm4c\" (UniqueName: \"kubernetes.io/projected/7f2a9bae-7ad2-45c5-a293-6090633cb6c7-kube-api-access-gjm4c\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:29.025220 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.025188 4227 generic.go:356] "Generic (PLEG): container finished" podID="7f2a9bae-7ad2-45c5-a293-6090633cb6c7" containerID="a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747" exitCode=0 Nov 03 21:27:29.025394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.025234 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" Nov 03 21:27:29.025394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.025285 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" event={"ID":"7f2a9bae-7ad2-45c5-a293-6090633cb6c7","Type":"ContainerDied","Data":"a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747"} Nov 03 21:27:29.025394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.025330 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc" event={"ID":"7f2a9bae-7ad2-45c5-a293-6090633cb6c7","Type":"ContainerDied","Data":"d7c872e29d5d38c2cb863d1ef1e0fa207e06314d39cccfa3981dbd269ecdcb9d"} Nov 03 21:27:29.025394 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.025350 4227 scope.go:122] "RemoveContainer" containerID="a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747" Nov 03 21:27:29.033181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.033149 4227 scope.go:122] "RemoveContainer" containerID="a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747" Nov 03 21:27:29.033460 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:27:29.033439 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747\": container with ID starting with a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747 not found: ID does not exist" containerID="a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747" Nov 03 21:27:29.033519 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.033467 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747"} err="failed to get container status \"a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747\": rpc error: code = NotFound desc = could not find container \"a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747\": container with ID starting with a5c7c9982d97c5c7eb265e638c97883d4f75ffbe662b54574a00175e43870747 not found: ID does not exist" Nov 03 21:27:29.178197 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.178150 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc"] Nov 03 21:27:29.223862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:29.223831 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d9f39a/simulator-single-main-d9f39a-kserve-8698db74b8-j9wgc"] Nov 03 21:27:30.898041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:27:30.898010 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2a9bae-7ad2-45c5-a293-6090633cb6c7" path="/var/lib/kubelet/pods/7f2a9bae-7ad2-45c5-a293-6090633cb6c7/volumes" Nov 03 21:27:56.081439 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:27:56.081403 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:28:01.925335 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:01.925291 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls"] Nov 03 21:28:01.925913 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:01.925588 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" podUID="c88e2c41-c43e-4d07-bcee-ca2a13039ab7" containerName="manager" containerID="cri-o://34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1" gracePeriod=10 Nov 03 21:28:02.164082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.164057 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 21:28:02.164570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.164545 4227 generic.go:356] "Generic (PLEG): container finished" podID="c88e2c41-c43e-4d07-bcee-ca2a13039ab7" containerID="34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1" exitCode=0 Nov 03 21:28:02.164672 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.164614 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" event={"ID":"c88e2c41-c43e-4d07-bcee-ca2a13039ab7","Type":"ContainerDied","Data":"34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1"} Nov 03 21:28:02.164672 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.164663 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" event={"ID":"c88e2c41-c43e-4d07-bcee-ca2a13039ab7","Type":"ContainerDied","Data":"29fec6e4a04fd2d7944a08e13a6e517c136f76113fe31329d6bc4d6be3d02651"} Nov 03 21:28:02.164747 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.164688 4227 scope.go:122] "RemoveContainer" containerID="34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1" Nov 03 21:28:02.176201 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.176178 4227 scope.go:122] "RemoveContainer" containerID="34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1" Nov 03 21:28:02.176468 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:28:02.176439 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1\": container with ID starting with 34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1 not found: ID does not exist" containerID="34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1" Nov 03 21:28:02.176533 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.176473 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1"} err="failed to get container status \"34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1\": rpc error: code = NotFound desc = could not find container \"34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1\": container with ID starting with 34af588fb22056e764c20b61e874176ce7696a497f42f367b54e167fe2efe8a1 not found: ID does not exist" Nov 03 21:28:02.289343 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.289312 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-extensions-socket-volume\") pod \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " Nov 03 21:28:02.289531 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.289355 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-kube-api-access-4sqls\" (UniqueName: \"kubernetes.io/projected/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-kube-api-access-4sqls\") pod \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\" (UID: \"c88e2c41-c43e-4d07-bcee-ca2a13039ab7\") " Nov 03 21:28:02.289814 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.289780 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-extensions-socket-volume" pod "c88e2c41-c43e-4d07-bcee-ca2a13039ab7" (UID: "c88e2c41-c43e-4d07-bcee-ca2a13039ab7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:02.292527 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.292501 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-kube-api-access-4sqls" pod "c88e2c41-c43e-4d07-bcee-ca2a13039ab7" (UID: "c88e2c41-c43e-4d07-bcee-ca2a13039ab7"). InnerVolumeSpecName "kube-api-access-4sqls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:02.390205 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.390174 4227 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-extensions-socket-volume\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:02.390205 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:02.390202 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sqls\" (UniqueName: \"kubernetes.io/projected/c88e2c41-c43e-4d07-bcee-ca2a13039ab7-kube-api-access-4sqls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:03.169206 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:03.169118 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls" Nov 03 21:28:03.369072 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:03.369036 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls"] Nov 03 21:28:03.419027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:03.418997 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7g8ls"] Nov 03 21:28:04.898213 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:04.898137 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88e2c41-c43e-4d07-bcee-ca2a13039ab7" path="/var/lib/kubelet/pods/c88e2c41-c43e-4d07-bcee-ca2a13039ab7/volumes" Nov 03 21:28:26.080832 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:28:26.080797 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:28:53.366607 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:53.366577 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:28:53.377553 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:28:53.377533 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:28:56.080074 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:28:56.080033 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:29:26.079910 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:29:26.079869 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:29:32.939653 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.939612 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf"] Nov 03 21:29:32.940186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.940081 4227 cpu_manager.go:414] "containerMap: removing container" podUID="7f2a9bae-7ad2-45c5-a293-6090633cb6c7" containerName="main" Nov 03 21:29:32.940186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.940099 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2a9bae-7ad2-45c5-a293-6090633cb6c7" containerName="main" Nov 03 21:29:32.940186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.940144 4227 cpu_manager.go:414] "containerMap: removing container" podUID="c88e2c41-c43e-4d07-bcee-ca2a13039ab7" containerName="manager" Nov 03 21:29:32.940186 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.940151 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88e2c41-c43e-4d07-bcee-ca2a13039ab7" containerName="manager" Nov 03 21:29:32.940435 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.940244 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2a9bae-7ad2-45c5-a293-6090633cb6c7" containerName="main" Nov 03 21:29:32.940435 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.940266 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88e2c41-c43e-4d07-bcee-ca2a13039ab7" containerName="manager" Nov 03 21:29:32.943051 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:32.943029 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.038272 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.038237 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.038272 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.038267 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.038507 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.038267 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:29:33.038507 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.038405 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:29:33.039785 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.039766 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf"] Nov 03 21:29:33.041785 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.041764 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.041880 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.041823 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2xj\" (UniqueName: \"kubernetes.io/projected/2adc4953-3186-4f54-9af3-446dfb38bf41-kube-api-access-xs2xj\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.041945 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.041916 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.143250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.143212 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2xj\" (UniqueName: \"kubernetes.io/projected/2adc4953-3186-4f54-9af3-446dfb38bf41-kube-api-access-xs2xj\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.143250 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.143253 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.143511 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.143298 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.143888 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.143866 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.145561 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.145542 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.196510 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.196437 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2xj\" (UniqueName: \"kubernetes.io/projected/2adc4953-3186-4f54-9af3-446dfb38bf41-kube-api-access-xs2xj\") pod \"maas-api-revoke-sub-d576e68b-qcvjf\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.255521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.255471 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:33.619886 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:33.619805 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf"] Nov 03 21:29:33.627629 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:29:33.627599 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adc4953_3186_4f54_9af3_446dfb38bf41.slice/crio-38946541fd3e6707f5d493ec6766ef1551b99c5dbbb828d58a6f3aac8f754053 WatchSource:0}: Error finding container 38946541fd3e6707f5d493ec6766ef1551b99c5dbbb828d58a6f3aac8f754053: Status 404 returned error can't find the container with id 38946541fd3e6707f5d493ec6766ef1551b99c5dbbb828d58a6f3aac8f754053 Nov 03 21:29:34.532920 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:34.532887 4227 generic.go:356] "Generic (PLEG): container finished" podID="2adc4953-3186-4f54-9af3-446dfb38bf41" containerID="e6e035c0149e8b2c96796460fae5d641125820ab26bec2167510f45d171a7280" exitCode=0 Nov 03 21:29:34.533314 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:34.532967 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" event={"ID":"2adc4953-3186-4f54-9af3-446dfb38bf41","Type":"ContainerDied","Data":"e6e035c0149e8b2c96796460fae5d641125820ab26bec2167510f45d171a7280"} Nov 03 21:29:34.533314 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:34.533003 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" event={"ID":"2adc4953-3186-4f54-9af3-446dfb38bf41","Type":"ContainerStarted","Data":"38946541fd3e6707f5d493ec6766ef1551b99c5dbbb828d58a6f3aac8f754053"} Nov 03 21:29:35.660876 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.660853 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:35.766465 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.766426 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-openshift-ca\") pod \"2adc4953-3186-4f54-9af3-446dfb38bf41\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " Nov 03 21:29:35.766695 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.766511 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adc4953-3186-4f54-9af3-446dfb38bf41-kube-api-access-xs2xj\" (UniqueName: \"kubernetes.io/projected/2adc4953-3186-4f54-9af3-446dfb38bf41-kube-api-access-xs2xj\") pod \"2adc4953-3186-4f54-9af3-446dfb38bf41\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " Nov 03 21:29:35.766695 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.766544 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-xks-ca\") pod \"2adc4953-3186-4f54-9af3-446dfb38bf41\" (UID: \"2adc4953-3186-4f54-9af3-446dfb38bf41\") " Nov 03 21:29:35.767017 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.766983 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-openshift-ca" pod "2adc4953-3186-4f54-9af3-446dfb38bf41" (UID: "2adc4953-3186-4f54-9af3-446dfb38bf41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:35.768975 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.768946 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adc4953-3186-4f54-9af3-446dfb38bf41-kube-api-access-xs2xj" pod "2adc4953-3186-4f54-9af3-446dfb38bf41" (UID: "2adc4953-3186-4f54-9af3-446dfb38bf41"). InnerVolumeSpecName "kube-api-access-xs2xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:35.769352 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.769063 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-xks-ca" pod "2adc4953-3186-4f54-9af3-446dfb38bf41" (UID: "2adc4953-3186-4f54-9af3-446dfb38bf41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:29:35.867862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.867775 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-openshift-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:35.867862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.867809 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2xj\" (UniqueName: \"kubernetes.io/projected/2adc4953-3186-4f54-9af3-446dfb38bf41-kube-api-access-xs2xj\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:35.867862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:35.867822 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adc4953-3186-4f54-9af3-446dfb38bf41-maas-api-xks-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:36.542497 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:36.542465 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" Nov 03 21:29:36.542680 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:36.542491 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf" event={"ID":"2adc4953-3186-4f54-9af3-446dfb38bf41","Type":"ContainerDied","Data":"38946541fd3e6707f5d493ec6766ef1551b99c5dbbb828d58a6f3aac8f754053"} Nov 03 21:29:36.542680 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:29:36.542544 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38946541fd3e6707f5d493ec6766ef1551b99c5dbbb828d58a6f3aac8f754053" Nov 03 21:29:56.081607 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:29:56.081570 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:30:00.173185 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.172494 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x"] Nov 03 21:30:00.180182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.173093 4227 cpu_manager.go:414] "containerMap: removing container" podUID="2adc4953-3186-4f54-9af3-446dfb38bf41" containerName="revoke-keys" Nov 03 21:30:00.180182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.176211 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2adc4953-3186-4f54-9af3-446dfb38bf41" containerName="revoke-keys" Nov 03 21:30:00.180182 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.176395 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="2adc4953-3186-4f54-9af3-446dfb38bf41" containerName="revoke-keys" Nov 03 21:30:00.215656 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.215621 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:00.231231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.231197 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x"] Nov 03 21:30:00.301569 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.301520 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:30:00.321109 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.320717 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwvhh\" (UniqueName: \"kubernetes.io/projected/4104e561-edf7-41ae-9538-25d71c9f8b19-kube-api-access-xwvhh\") pod \"maas-api-key-cleanup-29895690-bp92x\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:00.321109 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.320848 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4104e561-edf7-41ae-9538-25d71c9f8b19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-bp92x\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:00.352419 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.352369 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:30:00.352790 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.352767 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:30:00.421669 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.421625 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4104e561-edf7-41ae-9538-25d71c9f8b19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-bp92x\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:00.422020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.421999 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwvhh\" (UniqueName: \"kubernetes.io/projected/4104e561-edf7-41ae-9538-25d71c9f8b19-kube-api-access-xwvhh\") pod \"maas-api-key-cleanup-29895690-bp92x\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:00.422848 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.422817 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4104e561-edf7-41ae-9538-25d71c9f8b19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-bp92x\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:00.486232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.485960 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwvhh\" (UniqueName: \"kubernetes.io/projected/4104e561-edf7-41ae-9538-25d71c9f8b19-kube-api-access-xwvhh\") pod \"maas-api-key-cleanup-29895690-bp92x\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:00.539963 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:00.539921 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:01.134463 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:01.134137 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x"] Nov 03 21:30:01.682543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:01.682496 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" event={"ID":"4104e561-edf7-41ae-9538-25d71c9f8b19","Type":"ContainerStarted","Data":"d2659791c76878dca9cc1c1ebba0c9d802be1a43e818dcd03aca84565e1b0ede"} Nov 03 21:30:01.683057 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:01.682555 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" event={"ID":"4104e561-edf7-41ae-9538-25d71c9f8b19","Type":"ContainerStarted","Data":"8372f9a0ab098b9d153c7906350ea85806609f2bfbb0d2583d71d9c84985966e"} Nov 03 21:30:01.791338 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:01.791281 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" podStartSLOduration=1.7912679040000001 podStartE2EDuration="1.791267904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:01.789592067 +0000 UTC m=+3669.026569940" watchObservedRunningTime="2026-11-03 21:30:01.791267904 +0000 UTC m=+3669.028245778" Nov 03 21:30:02.687950 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:02.687913 4227 generic.go:356] "Generic (PLEG): container finished" podID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerID="d2659791c76878dca9cc1c1ebba0c9d802be1a43e818dcd03aca84565e1b0ede" exitCode=60 Nov 03 21:30:02.688425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:02.688006 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" event={"ID":"4104e561-edf7-41ae-9538-25d71c9f8b19","Type":"ContainerDied","Data":"d2659791c76878dca9cc1c1ebba0c9d802be1a43e818dcd03aca84565e1b0ede"} Nov 03 21:30:02.688425 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:02.688215 4227 scope.go:122] "RemoveContainer" containerID="d2659791c76878dca9cc1c1ebba0c9d802be1a43e818dcd03aca84565e1b0ede" Nov 03 21:30:03.693771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:03.693734 4227 generic.go:356] "Generic (PLEG): container finished" podID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerID="95dc10f69c063543f7c86e9bace05020de82b427a7427c86e80f030571fdaec8" exitCode=60 Nov 03 21:30:03.694274 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:03.693816 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" event={"ID":"4104e561-edf7-41ae-9538-25d71c9f8b19","Type":"ContainerDied","Data":"95dc10f69c063543f7c86e9bace05020de82b427a7427c86e80f030571fdaec8"} Nov 03 21:30:03.694274 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:03.693872 4227 scope.go:122] "RemoveContainer" containerID="d2659791c76878dca9cc1c1ebba0c9d802be1a43e818dcd03aca84565e1b0ede" Nov 03 21:30:03.694274 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:03.694181 4227 scope.go:122] "RemoveContainer" containerID="95dc10f69c063543f7c86e9bace05020de82b427a7427c86e80f030571fdaec8" Nov 03 21:30:03.694468 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:30:03.694445 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-bp92x_odh-ai-gateway-infra(4104e561-edf7-41ae-9538-25d71c9f8b19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" Nov 03 21:30:04.699826 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:04.699798 4227 scope.go:122] "RemoveContainer" containerID="95dc10f69c063543f7c86e9bace05020de82b427a7427c86e80f030571fdaec8" Nov 03 21:30:04.700278 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:30:04.700080 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-bp92x_odh-ai-gateway-infra(4104e561-edf7-41ae-9538-25d71c9f8b19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" Nov 03 21:30:14.897735 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:14.897706 4227 scope.go:122] "RemoveContainer" containerID="95dc10f69c063543f7c86e9bace05020de82b427a7427c86e80f030571fdaec8" Nov 03 21:30:15.744826 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:15.744793 4227 generic.go:356] "Generic (PLEG): container finished" podID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerID="23f635527468593330a3f44cb449b0e9625e743dea0f5569f7b2ba3d3b6c6f19" exitCode=60 Nov 03 21:30:15.744975 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:15.744860 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" event={"ID":"4104e561-edf7-41ae-9538-25d71c9f8b19","Type":"ContainerDied","Data":"23f635527468593330a3f44cb449b0e9625e743dea0f5569f7b2ba3d3b6c6f19"} Nov 03 21:30:15.744975 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:15.744914 4227 scope.go:122] "RemoveContainer" containerID="95dc10f69c063543f7c86e9bace05020de82b427a7427c86e80f030571fdaec8" Nov 03 21:30:15.745232 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:15.745215 4227 scope.go:122] "RemoveContainer" containerID="23f635527468593330a3f44cb449b0e9625e743dea0f5569f7b2ba3d3b6c6f19" Nov 03 21:30:15.745443 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:30:15.745422 4227 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-bp92x_odh-ai-gateway-infra(4104e561-edf7-41ae-9538-25d71c9f8b19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" Nov 03 21:30:16.859566 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:16.859536 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x"] Nov 03 21:30:17.001967 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.001943 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:17.065608 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.065584 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4104e561-edf7-41ae-9538-25d71c9f8b19-kube-api-access-xwvhh\" (UniqueName: \"kubernetes.io/projected/4104e561-edf7-41ae-9538-25d71c9f8b19-kube-api-access-xwvhh\") pod \"4104e561-edf7-41ae-9538-25d71c9f8b19\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " Nov 03 21:30:17.065725 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.065705 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4104e561-edf7-41ae-9538-25d71c9f8b19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4104e561-edf7-41ae-9538-25d71c9f8b19-maas-api-service-ca\") pod \"4104e561-edf7-41ae-9538-25d71c9f8b19\" (UID: \"4104e561-edf7-41ae-9538-25d71c9f8b19\") " Nov 03 21:30:17.066032 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.066008 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4104e561-edf7-41ae-9538-25d71c9f8b19-maas-api-service-ca" pod "4104e561-edf7-41ae-9538-25d71c9f8b19" (UID: "4104e561-edf7-41ae-9538-25d71c9f8b19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:17.067500 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.067476 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4104e561-edf7-41ae-9538-25d71c9f8b19-kube-api-access-xwvhh" pod "4104e561-edf7-41ae-9538-25d71c9f8b19" (UID: "4104e561-edf7-41ae-9538-25d71c9f8b19"). InnerVolumeSpecName "kube-api-access-xwvhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:17.166999 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.166965 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4104e561-edf7-41ae-9538-25d71c9f8b19-maas-api-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:17.167142 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.167003 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwvhh\" (UniqueName: \"kubernetes.io/projected/4104e561-edf7-41ae-9538-25d71c9f8b19-kube-api-access-xwvhh\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:17.755335 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.755300 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" event={"ID":"4104e561-edf7-41ae-9538-25d71c9f8b19","Type":"ContainerDied","Data":"8372f9a0ab098b9d153c7906350ea85806609f2bfbb0d2583d71d9c84985966e"} Nov 03 21:30:17.755335 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.755318 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x" Nov 03 21:30:17.755564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.755357 4227 scope.go:122] "RemoveContainer" containerID="23f635527468593330a3f44cb449b0e9625e743dea0f5569f7b2ba3d3b6c6f19" Nov 03 21:30:17.906317 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.906282 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x"] Nov 03 21:30:17.911623 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:17.911597 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-bp92x"] Nov 03 21:30:18.898273 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:18.898240 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" path="/var/lib/kubelet/pods/4104e561-edf7-41ae-9538-25d71c9f8b19/volumes" Nov 03 21:30:26.080291 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:30:26.080247 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:30:28.676890 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.676848 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g"] Nov 03 21:30:28.677437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.677386 4227 cpu_manager.go:414] "containerMap: removing container" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:28.677437 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.677406 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:28.677556 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.677440 4227 cpu_manager.go:414] "containerMap: removing container" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:28.677556 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.677449 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:28.677664 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.677569 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:28.677664 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.677586 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:28.679793 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.679771 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.776094 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.776064 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pll\" (UniqueName: \"kubernetes.io/projected/18ba762b-9622-4ba2-a6b7-b5ada892dba4-kube-api-access-j2pll\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.776272 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.776103 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.776272 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.776233 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.776353 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.776301 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ba762b-9622-4ba2-a6b7-b5ada892dba4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.778498 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.778460 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6d8154-serving-cert\"" Nov 03 21:30:28.778498 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.778477 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:30:28.778711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.778496 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:30:28.778711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.778539 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:30:28.778711 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.778580 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nqbhl\"" Nov 03 21:30:28.779976 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.779955 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g"] Nov 03 21:30:28.876751 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.876719 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.876914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.876773 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.876914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.876825 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ba762b-9622-4ba2-a6b7-b5ada892dba4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.876914 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:30:28.876873 4227 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-serving-cert: secret "maas-api-e2e-worker-single-main-6d8154-serving-cert" not found Nov 03 21:30:28.876914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.876884 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pll\" (UniqueName: \"kubernetes.io/projected/18ba762b-9622-4ba2-a6b7-b5ada892dba4-kube-api-access-j2pll\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.877137 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:30:28.876942 4227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls podName:18ba762b-9622-4ba2-a6b7-b5ada892dba4 nodeName:}" failed. No retries permitted until 2026-11-03 21:30:29.376926676 +0000 UTC m=+3696.613904528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls") pod "maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" (UID: "18ba762b-9622-4ba2-a6b7-b5ada892dba4") : secret "maas-api-e2e-worker-single-main-6d8154-serving-cert" not found Nov 03 21:30:28.877620 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.877595 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ba762b-9622-4ba2-a6b7-b5ada892dba4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.879311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.879294 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:28.931024 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:28.930942 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pll\" (UniqueName: \"kubernetes.io/projected/18ba762b-9622-4ba2-a6b7-b5ada892dba4-kube-api-access-j2pll\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:29.382386 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:29.382299 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:29.384643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:29.384620 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:29.590990 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:29.590953 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:29.954091 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:29.954047 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g"] Nov 03 21:30:29.961588 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:30:29.961557 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ba762b_9622_4ba2_a6b7_b5ada892dba4.slice/crio-6d2c1183961fa5561684db534104a2e019c6ef21de9b6caf08266e1a707a0d57 WatchSource:0}: Error finding container 6d2c1183961fa5561684db534104a2e019c6ef21de9b6caf08266e1a707a0d57: Status 404 returned error can't find the container with id 6d2c1183961fa5561684db534104a2e019c6ef21de9b6caf08266e1a707a0d57 Nov 03 21:30:29.962994 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:29.962973 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:30:30.818002 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:30.817624 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" event={"ID":"18ba762b-9622-4ba2-a6b7-b5ada892dba4","Type":"ContainerStarted","Data":"6d2c1183961fa5561684db534104a2e019c6ef21de9b6caf08266e1a707a0d57"} Nov 03 21:30:31.823288 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:31.823245 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" event={"ID":"18ba762b-9622-4ba2-a6b7-b5ada892dba4","Type":"ContainerStarted","Data":"d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513"} Nov 03 21:30:31.823781 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:31.823369 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:31.930019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:31.929961 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" podStartSLOduration=3.075318158 podStartE2EDuration="3.929946637s" totalImagesPullingTime="854.628479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:31.92987536 +0000 UTC m=+3699.166853233" watchObservedRunningTime="2026-11-03 21:30:31.929946637 +0000 UTC m=+3699.166924507" Nov 03 21:30:37.833802 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:37.833771 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:30:50.582171 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.582117 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8"] Nov 03 21:30:50.582594 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.582506 4227 cpu_manager.go:414] "containerMap: removing container" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:50.582594 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.582519 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:50.582594 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.582589 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="4104e561-edf7-41ae-9538-25d71c9f8b19" containerName="cleanup" Nov 03 21:30:50.585760 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.585730 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.589691 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.589664 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tmp-dir\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.589811 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.589714 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-model-cache\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.589811 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.589785 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tls-certs\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.589896 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.589823 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-home\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.589896 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.589869 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-747g5\" (UniqueName: \"kubernetes.io/projected/fbea1952-af95-4ba7-8c72-2574d46a3e7d-kube-api-access-747g5\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.590023 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.590001 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-dshm\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.683268 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.683235 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6d8154\"/\"kube-root-ca.crt\"" Nov 03 21:30:50.683458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.683274 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6d8154\"/\"simulator-single-main-6d8154-kserve-self-signed-certs\"" Nov 03 21:30:50.683458 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.683385 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6d8154\"/\"openshift-service-ca.crt\"" Nov 03 21:30:50.683545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.683460 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6d8154\"/\"default-dockercfg-v4492\"" Nov 03 21:30:50.684869 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.684850 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8"] Nov 03 21:30:50.690882 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.690866 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747g5\" (UniqueName: \"kubernetes.io/projected/fbea1952-af95-4ba7-8c72-2574d46a3e7d-kube-api-access-747g5\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.690934 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.690896 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-dshm\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.690934 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.690924 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tmp-dir\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.691054 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.691036 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-model-cache\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.691093 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.691071 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tls-certs\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.691145 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.691103 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-home\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.691335 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.691317 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tmp-dir\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.691412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.691396 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-model-cache\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.691503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.691483 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-home\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.693134 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.693106 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-dshm\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.694084 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.694059 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tls-certs\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.748258 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.748221 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-747g5\" (UniqueName: \"kubernetes.io/projected/fbea1952-af95-4ba7-8c72-2574d46a3e7d-kube-api-access-747g5\") pod \"simulator-single-main-6d8154-kserve-84fddf599d-87lb8\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:50.908646 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:50.908618 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:51.281405 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:51.281318 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8"] Nov 03 21:30:51.288678 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:30:51.288651 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbea1952_af95_4ba7_8c72_2574d46a3e7d.slice/crio-8a0ab291c1ff6bd94cf0d214c40aec6e4b04ca55e372321b79b601183387438f WatchSource:0}: Error finding container 8a0ab291c1ff6bd94cf0d214c40aec6e4b04ca55e372321b79b601183387438f: Status 404 returned error can't find the container with id 8a0ab291c1ff6bd94cf0d214c40aec6e4b04ca55e372321b79b601183387438f Nov 03 21:30:51.908434 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:51.908385 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" event={"ID":"fbea1952-af95-4ba7-8c72-2574d46a3e7d","Type":"ContainerStarted","Data":"0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04"} Nov 03 21:30:51.908434 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:51.908441 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" event={"ID":"fbea1952-af95-4ba7-8c72-2574d46a3e7d","Type":"ContainerStarted","Data":"8a0ab291c1ff6bd94cf0d214c40aec6e4b04ca55e372321b79b601183387438f"} Nov 03 21:30:51.908874 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:51.908801 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:30:52.017489 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:30:52.017430 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" podStartSLOduration=2.017412952 podStartE2EDuration="2.017412952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:52.016969112 +0000 UTC m=+3719.253946977" watchObservedRunningTime="2026-11-03 21:30:52.017412952 +0000 UTC m=+3719.254390829" Nov 03 21:30:56.080996 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:30:56.080953 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:31:00.908830 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:31:00.908796 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:31:00.925870 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:31:00.925835 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:31:00.960218 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:31:00.960175 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:31:26.082027 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:31:26.081990 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:31:51.633687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:31:51.633649 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8"] Nov 03 21:31:56.081763 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:31:56.081727 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:31:58.924097 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:31:58.924062 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g"] Nov 03 21:31:58.924771 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:31:58.924248 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" podUID="18ba762b-9622-4ba2-a6b7-b5ada892dba4" containerName="maas-api" containerID="cri-o://d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513" gracePeriod=30 Nov 03 21:32:00.168778 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.168752 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:32:00.198535 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.198501 4227 generic.go:356] "Generic (PLEG): container finished" podID="18ba762b-9622-4ba2-a6b7-b5ada892dba4" containerID="d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513" exitCode=0 Nov 03 21:32:00.198683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.198545 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" Nov 03 21:32:00.198683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.198566 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" event={"ID":"18ba762b-9622-4ba2-a6b7-b5ada892dba4","Type":"ContainerDied","Data":"d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513"} Nov 03 21:32:00.198683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.198637 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g" event={"ID":"18ba762b-9622-4ba2-a6b7-b5ada892dba4","Type":"ContainerDied","Data":"6d2c1183961fa5561684db534104a2e019c6ef21de9b6caf08266e1a707a0d57"} Nov 03 21:32:00.198683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.198663 4227 scope.go:122] "RemoveContainer" containerID="d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513" Nov 03 21:32:00.207044 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.207023 4227 scope.go:122] "RemoveContainer" containerID="d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513" Nov 03 21:32:00.207332 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:32:00.207311 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513\": container with ID starting with d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513 not found: ID does not exist" containerID="d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513" Nov 03 21:32:00.207418 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.207343 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513"} err="failed to get container status \"d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513\": rpc error: code = NotFound desc = could not find container \"d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513\": container with ID starting with d3dbe655713207faa60bc0fb0b963bfc547886c728370753176fcae1e8da8513 not found: ID does not exist" Nov 03 21:32:00.214674 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.214655 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls\") pod \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " Nov 03 21:32:00.214757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.214709 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ba762b-9622-4ba2-a6b7-b5ada892dba4-kube-api-access-j2pll\" (UniqueName: \"kubernetes.io/projected/18ba762b-9622-4ba2-a6b7-b5ada892dba4-kube-api-access-j2pll\") pod \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " Nov 03 21:32:00.214757 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.214742 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-metrics-certs\") pod \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " Nov 03 21:32:00.214831 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.214764 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ba762b-9622-4ba2-a6b7-b5ada892dba4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ba762b-9622-4ba2-a6b7-b5ada892dba4-openshift-service-ca\") pod \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\" (UID: \"18ba762b-9622-4ba2-a6b7-b5ada892dba4\") " Nov 03 21:32:00.215215 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.215136 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ba762b-9622-4ba2-a6b7-b5ada892dba4-openshift-service-ca" pod "18ba762b-9622-4ba2-a6b7-b5ada892dba4" (UID: "18ba762b-9622-4ba2-a6b7-b5ada892dba4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:00.216776 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.216751 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ba762b-9622-4ba2-a6b7-b5ada892dba4-kube-api-access-j2pll" pod "18ba762b-9622-4ba2-a6b7-b5ada892dba4" (UID: "18ba762b-9622-4ba2-a6b7-b5ada892dba4"). InnerVolumeSpecName "kube-api-access-j2pll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:00.216862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.216792 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls" pod "18ba762b-9622-4ba2-a6b7-b5ada892dba4" (UID: "18ba762b-9622-4ba2-a6b7-b5ada892dba4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:00.216862 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.216809 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-metrics-certs" pod "18ba762b-9622-4ba2-a6b7-b5ada892dba4" (UID: "18ba762b-9622-4ba2-a6b7-b5ada892dba4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:00.315973 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.315898 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2pll\" (UniqueName: \"kubernetes.io/projected/18ba762b-9622-4ba2-a6b7-b5ada892dba4-kube-api-access-j2pll\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:00.315973 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.315928 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:00.315973 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.315938 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ba762b-9622-4ba2-a6b7-b5ada892dba4-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:00.315973 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.315947 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ba762b-9622-4ba2-a6b7-b5ada892dba4-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:00.610412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.610331 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g"] Nov 03 21:32:00.613341 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.613317 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6d8154-6f4cb69874-h9d2g"] Nov 03 21:32:00.898843 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:00.898746 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ba762b-9622-4ba2-a6b7-b5ada892dba4" path="/var/lib/kubelet/pods/18ba762b-9622-4ba2-a6b7-b5ada892dba4/volumes" Nov 03 21:32:06.653468 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.653426 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" podUID="fbea1952-af95-4ba7-8c72-2574d46a3e7d" containerName="main" containerID="cri-o://0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04" gracePeriod=45 Nov 03 21:32:06.901175 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.901129 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:32:06.978512 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978423 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tmp-dir\") pod \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " Nov 03 21:32:06.978512 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978468 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tls-certs\") pod \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " Nov 03 21:32:06.978512 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978486 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-model-cache\") pod \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " Nov 03 21:32:06.978815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978534 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-home\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-home\") pod \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " Nov 03 21:32:06.978815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978573 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbea1952-af95-4ba7-8c72-2574d46a3e7d-kube-api-access-747g5\" (UniqueName: \"kubernetes.io/projected/fbea1952-af95-4ba7-8c72-2574d46a3e7d-kube-api-access-747g5\") pod \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " Nov 03 21:32:06.978815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978607 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-dshm\") pod \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\" (UID: \"fbea1952-af95-4ba7-8c72-2574d46a3e7d\") " Nov 03 21:32:06.978815 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978735 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tmp-dir" pod "fbea1952-af95-4ba7-8c72-2574d46a3e7d" (UID: "fbea1952-af95-4ba7-8c72-2574d46a3e7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:06.979018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978819 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-model-cache" pod "fbea1952-af95-4ba7-8c72-2574d46a3e7d" (UID: "fbea1952-af95-4ba7-8c72-2574d46a3e7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:06.979018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978839 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-home" pod "fbea1952-af95-4ba7-8c72-2574d46a3e7d" (UID: "fbea1952-af95-4ba7-8c72-2574d46a3e7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:06.979018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.978988 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:06.979018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.979002 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:06.979018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.979012 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:06.980896 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.980875 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-dshm" pod "fbea1952-af95-4ba7-8c72-2574d46a3e7d" (UID: "fbea1952-af95-4ba7-8c72-2574d46a3e7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:32:06.981002 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.980948 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbea1952-af95-4ba7-8c72-2574d46a3e7d-kube-api-access-747g5" pod "fbea1952-af95-4ba7-8c72-2574d46a3e7d" (UID: "fbea1952-af95-4ba7-8c72-2574d46a3e7d"). InnerVolumeSpecName "kube-api-access-747g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:06.981063 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:06.981012 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tls-certs" pod "fbea1952-af95-4ba7-8c72-2574d46a3e7d" (UID: "fbea1952-af95-4ba7-8c72-2574d46a3e7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:07.079773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.079738 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbea1952-af95-4ba7-8c72-2574d46a3e7d-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:07.079773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.079775 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-747g5\" (UniqueName: \"kubernetes.io/projected/fbea1952-af95-4ba7-8c72-2574d46a3e7d-kube-api-access-747g5\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:07.079996 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.079791 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbea1952-af95-4ba7-8c72-2574d46a3e7d-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:07.230632 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.230542 4227 generic.go:356] "Generic (PLEG): container finished" podID="fbea1952-af95-4ba7-8c72-2574d46a3e7d" containerID="0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04" exitCode=0 Nov 03 21:32:07.230632 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.230588 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" Nov 03 21:32:07.230821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.230627 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" event={"ID":"fbea1952-af95-4ba7-8c72-2574d46a3e7d","Type":"ContainerDied","Data":"0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04"} Nov 03 21:32:07.230821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.230669 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8" event={"ID":"fbea1952-af95-4ba7-8c72-2574d46a3e7d","Type":"ContainerDied","Data":"8a0ab291c1ff6bd94cf0d214c40aec6e4b04ca55e372321b79b601183387438f"} Nov 03 21:32:07.230821 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.230687 4227 scope.go:122] "RemoveContainer" containerID="0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04" Nov 03 21:32:07.240289 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.240269 4227 scope.go:122] "RemoveContainer" containerID="0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04" Nov 03 21:32:07.240793 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:32:07.240771 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04\": container with ID starting with 0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04 not found: ID does not exist" containerID="0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04" Nov 03 21:32:07.241000 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.240971 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04"} err="failed to get container status \"0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04\": rpc error: code = NotFound desc = could not find container \"0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04\": container with ID starting with 0dd6cba1a22edc40e9317739890d7eb47b212dbb73fc5307cfab286c184dda04 not found: ID does not exist" Nov 03 21:32:07.383238 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.383201 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8"] Nov 03 21:32:07.386467 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:07.386441 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-6d8154/simulator-single-main-6d8154-kserve-84fddf599d-87lb8"] Nov 03 21:32:08.898301 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:08.898264 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbea1952-af95-4ba7-8c72-2574d46a3e7d" path="/var/lib/kubelet/pods/fbea1952-af95-4ba7-8c72-2574d46a3e7d/volumes" Nov 03 21:32:17.053244 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.053206 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx"] Nov 03 21:32:17.053755 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.053652 4227 cpu_manager.go:414] "containerMap: removing container" podUID="18ba762b-9622-4ba2-a6b7-b5ada892dba4" containerName="maas-api" Nov 03 21:32:17.053755 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.053671 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ba762b-9622-4ba2-a6b7-b5ada892dba4" containerName="maas-api" Nov 03 21:32:17.053755 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.053697 4227 cpu_manager.go:414] "containerMap: removing container" podUID="fbea1952-af95-4ba7-8c72-2574d46a3e7d" containerName="main" Nov 03 21:32:17.053755 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.053705 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbea1952-af95-4ba7-8c72-2574d46a3e7d" containerName="main" Nov 03 21:32:17.053971 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.053808 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbea1952-af95-4ba7-8c72-2574d46a3e7d" containerName="main" Nov 03 21:32:17.053971 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.053829 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ba762b-9622-4ba2-a6b7-b5ada892dba4" containerName="maas-api" Nov 03 21:32:17.057749 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.057728 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.110952 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.110915 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx"] Nov 03 21:32:17.154693 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.154662 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 21:32:17.155134 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.155121 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 21:32:17.155257 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.155186 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9555c5d3-kserve-self-signed-certs\"" Nov 03 21:32:17.155257 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.155242 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mr542\"" Nov 03 21:32:17.166545 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.166516 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-home\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.166646 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.166561 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tmp-dir\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.166646 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.166636 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-dshm\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.166747 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.166732 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-model-cache\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.166798 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.166762 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qfj\" (UniqueName: \"kubernetes.io/projected/3edf04f3-3e41-4bd2-971c-4027583bdf5c-kube-api-access-n2qfj\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.166798 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.166793 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tls-certs\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.267457 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.267419 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tls-certs\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.267643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.267483 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-home\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.267643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.267519 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tmp-dir\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.267643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.267548 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-dshm\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.267643 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.267623 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-model-cache\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.267903 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.267658 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qfj\" (UniqueName: \"kubernetes.io/projected/3edf04f3-3e41-4bd2-971c-4027583bdf5c-kube-api-access-n2qfj\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.268055 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.268024 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tmp-dir\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.268055 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.268040 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-model-cache\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.268205 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.268140 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-home\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.269881 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.269859 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-dshm\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.270195 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.270151 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tls-certs\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.320748 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.320687 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qfj\" (UniqueName: \"kubernetes.io/projected/3edf04f3-3e41-4bd2-971c-4027583bdf5c-kube-api-access-n2qfj\") pod \"e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.377902 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.377877 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:17.766919 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:17.766879 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx"] Nov 03 21:32:17.773819 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:32:17.773792 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3edf04f3_3e41_4bd2_971c_4027583bdf5c.slice/crio-e5dd4ba4278d540d18f49b28155e5090556eade3a17449125abc4d93965143f6 WatchSource:0}: Error finding container e5dd4ba4278d540d18f49b28155e5090556eade3a17449125abc4d93965143f6: Status 404 returned error can't find the container with id e5dd4ba4278d540d18f49b28155e5090556eade3a17449125abc4d93965143f6 Nov 03 21:32:18.294641 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:18.294605 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" event={"ID":"3edf04f3-3e41-4bd2-971c-4027583bdf5c","Type":"ContainerStarted","Data":"205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034"} Nov 03 21:32:18.294999 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:18.294648 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" event={"ID":"3edf04f3-3e41-4bd2-971c-4027583bdf5c","Type":"ContainerStarted","Data":"e5dd4ba4278d540d18f49b28155e5090556eade3a17449125abc4d93965143f6"} Nov 03 21:32:18.378569 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:18.378534 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:18.402403 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:18.402352 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" podStartSLOduration=1.402339374 podStartE2EDuration="1.402339374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:18.401239098 +0000 UTC m=+3805.638216971" watchObservedRunningTime="2026-11-03 21:32:18.402339374 +0000 UTC m=+3805.639317248" Nov 03 21:32:26.083015 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:32:26.082976 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:32:27.378564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:27.378532 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:27.391030 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:27.391003 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:28.355560 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:28.355524 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:32:56.082196 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:32:56.082143 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:32:57.638295 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.638255 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt"] Nov 03 21:32:57.642928 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.642910 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.725914 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.725881 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-model-cache\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.726085 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.725922 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-dshm\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.726085 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.726002 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8r7\" (UniqueName: \"kubernetes.io/projected/467d3be7-ab35-4eab-b181-b76bb135d182-kube-api-access-5w8r7\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.726085 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.726072 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-home\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.726251 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.726119 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-tmp-dir\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.726251 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.726154 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467d3be7-ab35-4eab-b181-b76bb135d182-tls-certs\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.739575 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.739549 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9555c5d3-kserve-self-signed-certs\"" Nov 03 21:32:57.741137 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.741117 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt"] Nov 03 21:32:57.827229 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827192 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8r7\" (UniqueName: \"kubernetes.io/projected/467d3be7-ab35-4eab-b181-b76bb135d182-kube-api-access-5w8r7\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827229 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827228 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-home\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827511 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827271 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-tmp-dir\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827511 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827310 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467d3be7-ab35-4eab-b181-b76bb135d182-tls-certs\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827511 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827390 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-model-cache\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827511 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827421 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-dshm\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827767 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827743 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-home\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827842 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827774 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-tmp-dir\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.827901 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.827861 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-model-cache\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.829724 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.829700 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-dshm\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.829980 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.829960 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467d3be7-ab35-4eab-b181-b76bb135d182-tls-certs\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.883490 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.883465 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8r7\" (UniqueName: \"kubernetes.io/projected/467d3be7-ab35-4eab-b181-b76bb135d182-kube-api-access-5w8r7\") pod \"e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:57.953975 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:57.953944 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:32:58.339001 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:58.338923 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt"] Nov 03 21:32:58.346536 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:32:58.346501 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod467d3be7_ab35_4eab_b181_b76bb135d182.slice/crio-09182413ed73bfc9c2da1ce780fe7f1dd84d209808627479dbb8bd6fb10ef300 WatchSource:0}: Error finding container 09182413ed73bfc9c2da1ce780fe7f1dd84d209808627479dbb8bd6fb10ef300: Status 404 returned error can't find the container with id 09182413ed73bfc9c2da1ce780fe7f1dd84d209808627479dbb8bd6fb10ef300 Nov 03 21:32:58.471171 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:58.471131 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" event={"ID":"467d3be7-ab35-4eab-b181-b76bb135d182","Type":"ContainerStarted","Data":"2cfedfc4e8982b894f723ce9ca9820f900c8ccd80e96c273d453592851a2d440"} Nov 03 21:32:58.471342 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:58.471192 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" event={"ID":"467d3be7-ab35-4eab-b181-b76bb135d182","Type":"ContainerStarted","Data":"09182413ed73bfc9c2da1ce780fe7f1dd84d209808627479dbb8bd6fb10ef300"} Nov 03 21:32:58.578433 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:58.578379 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" podStartSLOduration=1.578364512 podStartE2EDuration="1.578364512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:58.57708644 +0000 UTC m=+3845.814064315" watchObservedRunningTime="2026-11-03 21:32:58.578364512 +0000 UTC m=+3845.815342380" Nov 03 21:32:58.954840 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:32:58.954807 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:33:07.954522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:07.954481 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:33:07.967134 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:07.967111 4227 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:33:08.548912 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:08.548878 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:33:26.081196 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:33:26.081139 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:33:28.027909 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.027877 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz"] Nov 03 21:33:28.031328 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.031311 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.084920 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.084889 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz"] Nov 03 21:33:28.129023 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.128992 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:33:28.129258 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.129239 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:33:28.129364 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.129282 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:33:28.129364 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.129305 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:33:28.141140 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.141118 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3351269f-df13-477f-a717-01507f976a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.141263 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.141155 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxm7k\" (UniqueName: \"kubernetes.io/projected/3351269f-df13-477f-a717-01507f976a00-kube-api-access-cxm7k\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.141363 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.141344 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3351269f-df13-477f-a717-01507f976a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.242430 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.242402 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3351269f-df13-477f-a717-01507f976a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.242543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.242450 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxm7k\" (UniqueName: \"kubernetes.io/projected/3351269f-df13-477f-a717-01507f976a00-kube-api-access-cxm7k\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.242608 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.242538 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3351269f-df13-477f-a717-01507f976a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.242996 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.242975 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3351269f-df13-477f-a717-01507f976a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.244717 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.244698 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3351269f-df13-477f-a717-01507f976a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.295375 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.295328 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxm7k\" (UniqueName: \"kubernetes.io/projected/3351269f-df13-477f-a717-01507f976a00-kube-api-access-cxm7k\") pod \"maas-api-revoke-sub-452b6d7b-p8lzz\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.343759 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.343740 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:28.706181 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:28.706133 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz"] Nov 03 21:33:28.714077 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:33:28.714048 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3351269f_df13_477f_a717_01507f976a00.slice/crio-a42e7651ee4f5f64fd00536fb9149bb1e5ac0ec5e6a438657ecc0f78399e9adb WatchSource:0}: Error finding container a42e7651ee4f5f64fd00536fb9149bb1e5ac0ec5e6a438657ecc0f78399e9adb: Status 404 returned error can't find the container with id a42e7651ee4f5f64fd00536fb9149bb1e5ac0ec5e6a438657ecc0f78399e9adb Nov 03 21:33:29.630212 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:29.630149 4227 generic.go:356] "Generic (PLEG): container finished" podID="3351269f-df13-477f-a717-01507f976a00" containerID="7ebdf051cbc8cd41c66bf1db395e50c1a57a28a04709ef724085d956a8c974a1" exitCode=0 Nov 03 21:33:29.630718 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:29.630219 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" event={"ID":"3351269f-df13-477f-a717-01507f976a00","Type":"ContainerDied","Data":"7ebdf051cbc8cd41c66bf1db395e50c1a57a28a04709ef724085d956a8c974a1"} Nov 03 21:33:29.630718 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:29.630281 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" event={"ID":"3351269f-df13-477f-a717-01507f976a00","Type":"ContainerStarted","Data":"a42e7651ee4f5f64fd00536fb9149bb1e5ac0ec5e6a438657ecc0f78399e9adb"} Nov 03 21:33:31.038298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.038267 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:31.218693 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.218647 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3351269f-df13-477f-a717-01507f976a00-kube-api-access-cxm7k\" (UniqueName: \"kubernetes.io/projected/3351269f-df13-477f-a717-01507f976a00-kube-api-access-cxm7k\") pod \"3351269f-df13-477f-a717-01507f976a00\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " Nov 03 21:33:31.218983 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.218956 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3351269f-df13-477f-a717-01507f976a00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3351269f-df13-477f-a717-01507f976a00-maas-api-xks-ca\") pod \"3351269f-df13-477f-a717-01507f976a00\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " Nov 03 21:33:31.219263 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.219241 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3351269f-df13-477f-a717-01507f976a00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3351269f-df13-477f-a717-01507f976a00-maas-api-openshift-ca\") pod \"3351269f-df13-477f-a717-01507f976a00\" (UID: \"3351269f-df13-477f-a717-01507f976a00\") " Nov 03 21:33:31.220652 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.220628 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3351269f-df13-477f-a717-01507f976a00-maas-api-openshift-ca" pod "3351269f-df13-477f-a717-01507f976a00" (UID: "3351269f-df13-477f-a717-01507f976a00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:33:31.227709 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.227678 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3351269f-df13-477f-a717-01507f976a00-maas-api-xks-ca" pod "3351269f-df13-477f-a717-01507f976a00" (UID: "3351269f-df13-477f-a717-01507f976a00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:33:31.228620 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.228596 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3351269f-df13-477f-a717-01507f976a00-kube-api-access-cxm7k" pod "3351269f-df13-477f-a717-01507f976a00" (UID: "3351269f-df13-477f-a717-01507f976a00"). InnerVolumeSpecName "kube-api-access-cxm7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:33:31.321275 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.321172 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxm7k\" (UniqueName: \"kubernetes.io/projected/3351269f-df13-477f-a717-01507f976a00-kube-api-access-cxm7k\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:31.321560 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.321537 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3351269f-df13-477f-a717-01507f976a00-maas-api-xks-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:31.321687 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.321672 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3351269f-df13-477f-a717-01507f976a00-maas-api-openshift-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:31.640359 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.640278 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" event={"ID":"3351269f-df13-477f-a717-01507f976a00","Type":"ContainerDied","Data":"a42e7651ee4f5f64fd00536fb9149bb1e5ac0ec5e6a438657ecc0f78399e9adb"} Nov 03 21:33:31.640359 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.640315 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz" Nov 03 21:33:31.640551 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:31.640328 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42e7651ee4f5f64fd00536fb9149bb1e5ac0ec5e6a438657ecc0f78399e9adb" Nov 03 21:33:40.731020 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:40.730988 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx"] Nov 03 21:33:53.392693 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:53.392662 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:33:53.403622 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:53.403600 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:33:55.750027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:55.749985 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" podUID="3edf04f3-3e41-4bd2-971c-4027583bdf5c" containerName="main" containerID="cri-o://205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034" gracePeriod=45 Nov 03 21:33:55.991897 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:55.991873 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:33:56.071314 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071237 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tls-certs\") pod \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " Nov 03 21:33:56.071461 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071313 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-home\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-home\") pod \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " Nov 03 21:33:56.071461 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071364 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tmp-dir\") pod \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " Nov 03 21:33:56.071461 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071414 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-dshm\") pod \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " Nov 03 21:33:56.071461 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071452 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edf04f3-3e41-4bd2-971c-4027583bdf5c-kube-api-access-n2qfj\" (UniqueName: \"kubernetes.io/projected/3edf04f3-3e41-4bd2-971c-4027583bdf5c-kube-api-access-n2qfj\") pod \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " Nov 03 21:33:56.071774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071554 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-model-cache\") pod \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\" (UID: \"3edf04f3-3e41-4bd2-971c-4027583bdf5c\") " Nov 03 21:33:56.071774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071611 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-home" pod "3edf04f3-3e41-4bd2-971c-4027583bdf5c" (UID: "3edf04f3-3e41-4bd2-971c-4027583bdf5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:56.071774 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071622 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tmp-dir" pod "3edf04f3-3e41-4bd2-971c-4027583bdf5c" (UID: "3edf04f3-3e41-4bd2-971c-4027583bdf5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:56.071953 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071812 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:56.071953 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071855 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:56.071953 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.071916 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-model-cache" pod "3edf04f3-3e41-4bd2-971c-4027583bdf5c" (UID: "3edf04f3-3e41-4bd2-971c-4027583bdf5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:56.073955 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.073930 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tls-certs" pod "3edf04f3-3e41-4bd2-971c-4027583bdf5c" (UID: "3edf04f3-3e41-4bd2-971c-4027583bdf5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:33:56.075693 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.075667 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edf04f3-3e41-4bd2-971c-4027583bdf5c-kube-api-access-n2qfj" pod "3edf04f3-3e41-4bd2-971c-4027583bdf5c" (UID: "3edf04f3-3e41-4bd2-971c-4027583bdf5c"). InnerVolumeSpecName "kube-api-access-n2qfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:33:56.077267 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.077242 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-dshm" pod "3edf04f3-3e41-4bd2-971c-4027583bdf5c" (UID: "3edf04f3-3e41-4bd2-971c-4027583bdf5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:33:56.083820 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:33:56.083792 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:33:56.173296 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.173266 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edf04f3-3e41-4bd2-971c-4027583bdf5c-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:56.173296 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.173296 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:56.173466 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.173311 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qfj\" (UniqueName: \"kubernetes.io/projected/3edf04f3-3e41-4bd2-971c-4027583bdf5c-kube-api-access-n2qfj\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:56.173466 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.173325 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edf04f3-3e41-4bd2-971c-4027583bdf5c-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:33:56.760961 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.760927 4227 generic.go:356] "Generic (PLEG): container finished" podID="3edf04f3-3e41-4bd2-971c-4027583bdf5c" containerID="205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034" exitCode=0 Nov 03 21:33:56.761378 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.760973 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" Nov 03 21:33:56.761378 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.761021 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" event={"ID":"3edf04f3-3e41-4bd2-971c-4027583bdf5c","Type":"ContainerDied","Data":"205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034"} Nov 03 21:33:56.761378 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.761067 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx" event={"ID":"3edf04f3-3e41-4bd2-971c-4027583bdf5c","Type":"ContainerDied","Data":"e5dd4ba4278d540d18f49b28155e5090556eade3a17449125abc4d93965143f6"} Nov 03 21:33:56.761378 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.761088 4227 scope.go:122] "RemoveContainer" containerID="205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034" Nov 03 21:33:56.770448 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.770426 4227 scope.go:122] "RemoveContainer" containerID="205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034" Nov 03 21:33:56.770693 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:33:56.770676 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034\": container with ID starting with 205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034 not found: ID does not exist" containerID="205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034" Nov 03 21:33:56.770744 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.770703 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034"} err="failed to get container status \"205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034\": rpc error: code = NotFound desc = could not find container \"205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034\": container with ID starting with 205839ae63deb61a5ab8a919d84891e0405eb8cf0683ddc3834c133f7fd35034 not found: ID does not exist" Nov 03 21:33:56.912869 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.912833 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx"] Nov 03 21:33:56.959522 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:56.959495 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9555c5d3-kserve-7dbfcf55bf-cv9wx"] Nov 03 21:33:58.897524 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:58.897495 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edf04f3-3e41-4bd2-971c-4027583bdf5c" path="/var/lib/kubelet/pods/3edf04f3-3e41-4bd2-971c-4027583bdf5c/volumes" Nov 03 21:33:59.727564 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:33:59.727519 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt"] Nov 03 21:34:03.838994 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.838949 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd"] Nov 03 21:34:03.839525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.839288 4227 cpu_manager.go:414] "containerMap: removing container" podUID="3edf04f3-3e41-4bd2-971c-4027583bdf5c" containerName="main" Nov 03 21:34:03.839525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.839299 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edf04f3-3e41-4bd2-971c-4027583bdf5c" containerName="main" Nov 03 21:34:03.839525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.839310 4227 cpu_manager.go:414] "containerMap: removing container" podUID="3351269f-df13-477f-a717-01507f976a00" containerName="revoke-keys" Nov 03 21:34:03.839525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.839316 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3351269f-df13-477f-a717-01507f976a00" containerName="revoke-keys" Nov 03 21:34:03.839525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.839386 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edf04f3-3e41-4bd2-971c-4027583bdf5c" containerName="main" Nov 03 21:34:03.839525 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.839393 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="3351269f-df13-477f-a717-01507f976a00" containerName="revoke-keys" Nov 03 21:34:03.842280 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.842254 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:03.894975 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.894939 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd"] Nov 03 21:34:03.939298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.939258 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:34:03.939298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.939271 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:34:03.939503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.939270 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:34:03.939503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.939346 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:34:03.943773 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.943748 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:03.943887 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.943819 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:03.943945 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:03.943902 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52cg\" (UniqueName: \"kubernetes.io/projected/31f71a28-de40-42ef-9c7b-6d04886a4f4f-kube-api-access-b52cg\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.045010 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.044974 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.045217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.045038 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.045217 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.045078 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52cg\" (UniqueName: \"kubernetes.io/projected/31f71a28-de40-42ef-9c7b-6d04886a4f4f-kube-api-access-b52cg\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.045787 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.045767 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.047299 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.047281 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.098835 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.098746 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52cg\" (UniqueName: \"kubernetes.io/projected/31f71a28-de40-42ef-9c7b-6d04886a4f4f-kube-api-access-b52cg\") pod \"maas-api-revoke-sub-3e31a506-sznqd\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.156354 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.156325 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:04.523345 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.523304 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd"] Nov 03 21:34:04.530778 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:34:04.530746 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f71a28_de40_42ef_9c7b_6d04886a4f4f.slice/crio-8f8dbdaf6b9679ac86a7e14ca6cd427db03a40084b1678d4a114a1bfed806e9d WatchSource:0}: Error finding container 8f8dbdaf6b9679ac86a7e14ca6cd427db03a40084b1678d4a114a1bfed806e9d: Status 404 returned error can't find the container with id 8f8dbdaf6b9679ac86a7e14ca6cd427db03a40084b1678d4a114a1bfed806e9d Nov 03 21:34:04.798111 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:04.798023 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" event={"ID":"31f71a28-de40-42ef-9c7b-6d04886a4f4f","Type":"ContainerStarted","Data":"8f8dbdaf6b9679ac86a7e14ca6cd427db03a40084b1678d4a114a1bfed806e9d"} Nov 03 21:34:05.821060 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:05.821022 4227 generic.go:356] "Generic (PLEG): container finished" podID="31f71a28-de40-42ef-9c7b-6d04886a4f4f" containerID="f0faea4e6abf0eb6134ef0cd59ba2ac4875d1e46ed19162f3a3bc1b5ec22de74" exitCode=0 Nov 03 21:34:05.821696 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:05.821524 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" event={"ID":"31f71a28-de40-42ef-9c7b-6d04886a4f4f","Type":"ContainerDied","Data":"f0faea4e6abf0eb6134ef0cd59ba2ac4875d1e46ed19162f3a3bc1b5ec22de74"} Nov 03 21:34:06.956969 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:06.956937 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:07.022543 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.022514 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f71a28-de40-42ef-9c7b-6d04886a4f4f-kube-api-access-b52cg\" (UniqueName: \"kubernetes.io/projected/31f71a28-de40-42ef-9c7b-6d04886a4f4f-kube-api-access-b52cg\") pod \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " Nov 03 21:34:07.022677 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.022551 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-openshift-ca\") pod \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " Nov 03 21:34:07.022677 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.022600 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-xks-ca\") pod \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\" (UID: \"31f71a28-de40-42ef-9c7b-6d04886a4f4f\") " Nov 03 21:34:07.022956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.022933 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-openshift-ca" pod "31f71a28-de40-42ef-9c7b-6d04886a4f4f" (UID: "31f71a28-de40-42ef-9c7b-6d04886a4f4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:34:07.024480 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.024460 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f71a28-de40-42ef-9c7b-6d04886a4f4f-kube-api-access-b52cg" pod "31f71a28-de40-42ef-9c7b-6d04886a4f4f" (UID: "31f71a28-de40-42ef-9c7b-6d04886a4f4f"). InnerVolumeSpecName "kube-api-access-b52cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:34:07.024602 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.024584 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-xks-ca" pod "31f71a28-de40-42ef-9c7b-6d04886a4f4f" (UID: "31f71a28-de40-42ef-9c7b-6d04886a4f4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:34:07.123648 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.123581 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-xks-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:07.123648 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.123608 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b52cg\" (UniqueName: \"kubernetes.io/projected/31f71a28-de40-42ef-9c7b-6d04886a4f4f-kube-api-access-b52cg\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:07.123648 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.123618 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f71a28-de40-42ef-9c7b-6d04886a4f4f-maas-api-openshift-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:07.830797 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.830760 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" Nov 03 21:34:07.830968 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.830760 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd" event={"ID":"31f71a28-de40-42ef-9c7b-6d04886a4f4f","Type":"ContainerDied","Data":"8f8dbdaf6b9679ac86a7e14ca6cd427db03a40084b1678d4a114a1bfed806e9d"} Nov 03 21:34:07.830968 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:07.830897 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f8dbdaf6b9679ac86a7e14ca6cd427db03a40084b1678d4a114a1bfed806e9d" Nov 03 21:34:14.746402 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:14.746337 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" podUID="467d3be7-ab35-4eab-b181-b76bb135d182" containerName="main" containerID="cri-o://2cfedfc4e8982b894f723ce9ca9820f900c8ccd80e96c273d453592851a2d440" gracePeriod=45 Nov 03 21:34:14.862331 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:14.862294 4227 generic.go:356] "Generic (PLEG): container finished" podID="467d3be7-ab35-4eab-b181-b76bb135d182" containerID="2cfedfc4e8982b894f723ce9ca9820f900c8ccd80e96c273d453592851a2d440" exitCode=0 Nov 03 21:34:14.862485 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:14.862362 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" event={"ID":"467d3be7-ab35-4eab-b181-b76bb135d182","Type":"ContainerDied","Data":"2cfedfc4e8982b894f723ce9ca9820f900c8ccd80e96c273d453592851a2d440"} Nov 03 21:34:14.996110 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:14.996080 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:34:15.090990 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.090871 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-model-cache\") pod \"467d3be7-ab35-4eab-b181-b76bb135d182\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " Nov 03 21:34:15.090990 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.090917 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-home\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-home\") pod \"467d3be7-ab35-4eab-b181-b76bb135d182\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " Nov 03 21:34:15.090990 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.090958 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467d3be7-ab35-4eab-b181-b76bb135d182-kube-api-access-5w8r7\" (UniqueName: \"kubernetes.io/projected/467d3be7-ab35-4eab-b181-b76bb135d182-kube-api-access-5w8r7\") pod \"467d3be7-ab35-4eab-b181-b76bb135d182\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " Nov 03 21:34:15.090990 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.090993 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/467d3be7-ab35-4eab-b181-b76bb135d182-tls-certs\" (UniqueName: \"kubernetes.io/secret/467d3be7-ab35-4eab-b181-b76bb135d182-tls-certs\") pod \"467d3be7-ab35-4eab-b181-b76bb135d182\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " Nov 03 21:34:15.091366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.091027 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-dshm\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-dshm\") pod \"467d3be7-ab35-4eab-b181-b76bb135d182\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " Nov 03 21:34:15.091366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.091053 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-tmp-dir\") pod \"467d3be7-ab35-4eab-b181-b76bb135d182\" (UID: \"467d3be7-ab35-4eab-b181-b76bb135d182\") " Nov 03 21:34:15.091366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.091233 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-model-cache" pod "467d3be7-ab35-4eab-b181-b76bb135d182" (UID: "467d3be7-ab35-4eab-b181-b76bb135d182"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:34:15.091366 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.091243 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-home" pod "467d3be7-ab35-4eab-b181-b76bb135d182" (UID: "467d3be7-ab35-4eab-b181-b76bb135d182"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:34:15.091589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.091373 4227 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-model-cache\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:15.091589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.091393 4227 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-home\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:15.091589 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.091433 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-tmp-dir" pod "467d3be7-ab35-4eab-b181-b76bb135d182" (UID: "467d3be7-ab35-4eab-b181-b76bb135d182"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:34:15.093241 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.093215 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-dshm" pod "467d3be7-ab35-4eab-b181-b76bb135d182" (UID: "467d3be7-ab35-4eab-b181-b76bb135d182"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:34:15.093317 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.093266 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467d3be7-ab35-4eab-b181-b76bb135d182-tls-certs" pod "467d3be7-ab35-4eab-b181-b76bb135d182" (UID: "467d3be7-ab35-4eab-b181-b76bb135d182"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:34:15.093361 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.093349 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467d3be7-ab35-4eab-b181-b76bb135d182-kube-api-access-5w8r7" pod "467d3be7-ab35-4eab-b181-b76bb135d182" (UID: "467d3be7-ab35-4eab-b181-b76bb135d182"). InnerVolumeSpecName "kube-api-access-5w8r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:34:15.192596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.192558 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8r7\" (UniqueName: \"kubernetes.io/projected/467d3be7-ab35-4eab-b181-b76bb135d182-kube-api-access-5w8r7\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:15.192596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.192590 4227 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/467d3be7-ab35-4eab-b181-b76bb135d182-tls-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:15.192596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.192600 4227 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-dshm\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:15.192596 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.192608 4227 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467d3be7-ab35-4eab-b181-b76bb135d182-tmp-dir\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:15.867469 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.867434 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" event={"ID":"467d3be7-ab35-4eab-b181-b76bb135d182","Type":"ContainerDied","Data":"09182413ed73bfc9c2da1ce780fe7f1dd84d209808627479dbb8bd6fb10ef300"} Nov 03 21:34:15.867911 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.867484 4227 scope.go:122] "RemoveContainer" containerID="2cfedfc4e8982b894f723ce9ca9820f900c8ccd80e96c273d453592851a2d440" Nov 03 21:34:15.867911 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:15.867442 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt" Nov 03 21:34:16.019927 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:16.019887 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt"] Nov 03 21:34:16.027460 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:16.027434 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9555c5d3-kserve-68b56b6b97-mpqnt"] Nov 03 21:34:16.899029 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:16.898994 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467d3be7-ab35-4eab-b181-b76bb135d182" path="/var/lib/kubelet/pods/467d3be7-ab35-4eab-b181-b76bb135d182/volumes" Nov 03 21:34:26.079780 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:34:26.079742 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:34:34.340957 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.340916 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64"] Nov 03 21:34:34.341413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.341269 4227 cpu_manager.go:414] "containerMap: removing container" podUID="31f71a28-de40-42ef-9c7b-6d04886a4f4f" containerName="revoke-keys" Nov 03 21:34:34.341413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.341282 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f71a28-de40-42ef-9c7b-6d04886a4f4f" containerName="revoke-keys" Nov 03 21:34:34.341413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.341303 4227 cpu_manager.go:414] "containerMap: removing container" podUID="467d3be7-ab35-4eab-b181-b76bb135d182" containerName="main" Nov 03 21:34:34.341413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.341307 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467d3be7-ab35-4eab-b181-b76bb135d182" containerName="main" Nov 03 21:34:34.341413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.341362 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f71a28-de40-42ef-9c7b-6d04886a4f4f" containerName="revoke-keys" Nov 03 21:34:34.341413 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.341373 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="467d3be7-ab35-4eab-b181-b76bb135d182" containerName="main" Nov 03 21:34:34.343683 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.343663 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.394896 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.394862 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:34:34.441151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.441128 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:34:34.441151 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.441148 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:34:34.441390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.441368 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\"" Nov 03 21:34:34.442766 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.442749 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64"] Nov 03 21:34:34.508770 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.508744 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.508917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.508799 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.508917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.508902 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj996\" (UniqueName: \"kubernetes.io/projected/0276489b-29b0-46da-a335-9cb35661ed1c-kube-api-access-qj996\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.609661 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.609599 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.609661 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.609636 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.609854 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.609666 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj996\" (UniqueName: \"kubernetes.io/projected/0276489b-29b0-46da-a335-9cb35661ed1c-kube-api-access-qj996\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.610367 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.610349 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.611973 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.611955 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.662409 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.662389 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj996\" (UniqueName: \"kubernetes.io/projected/0276489b-29b0-46da-a335-9cb35661ed1c-kube-api-access-qj996\") pod \"maas-api-revoke-sub-fdc19917-jtj64\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:34.955739 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:34.955712 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:35.319858 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:35.319776 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64"] Nov 03 21:34:35.326488 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:34:35.326459 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0276489b_29b0_46da_a335_9cb35661ed1c.slice/crio-1492abdeace1cdf2e2e12cbdf23eaa5e27195eb4282ba240e3590869e2b74276 WatchSource:0}: Error finding container 1492abdeace1cdf2e2e12cbdf23eaa5e27195eb4282ba240e3590869e2b74276: Status 404 returned error can't find the container with id 1492abdeace1cdf2e2e12cbdf23eaa5e27195eb4282ba240e3590869e2b74276 Nov 03 21:34:35.981412 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:35.981374 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" event={"ID":"0276489b-29b0-46da-a335-9cb35661ed1c","Type":"ContainerStarted","Data":"80e277ca98bdd18732cfacd3294ec991e11a27196d5fbb4c0e53f6d00fa614a1"} Nov 03 21:34:35.981809 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:35.981429 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" event={"ID":"0276489b-29b0-46da-a335-9cb35661ed1c","Type":"ContainerStarted","Data":"1492abdeace1cdf2e2e12cbdf23eaa5e27195eb4282ba240e3590869e2b74276"} Nov 03 21:34:36.079709 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:36.079679 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf"] Nov 03 21:34:36.080849 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:36.080829 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qcvjf"] Nov 03 21:34:36.125806 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:36.125758 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" podStartSLOduration=2.125746109 podStartE2EDuration="2.125746109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:36.088582535 +0000 UTC m=+3943.325560434" watchObservedRunningTime="2026-11-03 21:34:36.125746109 +0000 UTC m=+3943.362723982" Nov 03 21:34:36.898060 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:36.898029 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2adc4953-3186-4f54-9af3-446dfb38bf41" path="/var/lib/kubelet/pods/2adc4953-3186-4f54-9af3-446dfb38bf41/volumes" Nov 03 21:34:36.987127 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:36.987099 4227 generic.go:356] "Generic (PLEG): container finished" podID="0276489b-29b0-46da-a335-9cb35661ed1c" containerID="80e277ca98bdd18732cfacd3294ec991e11a27196d5fbb4c0e53f6d00fa614a1" exitCode=0 Nov 03 21:34:36.987501 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:36.987184 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" event={"ID":"0276489b-29b0-46da-a335-9cb35661ed1c","Type":"ContainerDied","Data":"80e277ca98bdd18732cfacd3294ec991e11a27196d5fbb4c0e53f6d00fa614a1"} Nov 03 21:34:38.119316 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.119292 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:38.241563 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.241531 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-xks-ca\") pod \"0276489b-29b0-46da-a335-9cb35661ed1c\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " Nov 03 21:34:38.241749 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.241594 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0276489b-29b0-46da-a335-9cb35661ed1c-kube-api-access-qj996\" (UniqueName: \"kubernetes.io/projected/0276489b-29b0-46da-a335-9cb35661ed1c-kube-api-access-qj996\") pod \"0276489b-29b0-46da-a335-9cb35661ed1c\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " Nov 03 21:34:38.241749 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.241692 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-openshift-ca\") pod \"0276489b-29b0-46da-a335-9cb35661ed1c\" (UID: \"0276489b-29b0-46da-a335-9cb35661ed1c\") " Nov 03 21:34:38.242116 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.242084 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-openshift-ca" pod "0276489b-29b0-46da-a335-9cb35661ed1c" (UID: "0276489b-29b0-46da-a335-9cb35661ed1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:34:38.243611 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.243586 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-xks-ca" pod "0276489b-29b0-46da-a335-9cb35661ed1c" (UID: "0276489b-29b0-46da-a335-9cb35661ed1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:34:38.243700 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.243659 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0276489b-29b0-46da-a335-9cb35661ed1c-kube-api-access-qj996" pod "0276489b-29b0-46da-a335-9cb35661ed1c" (UID: "0276489b-29b0-46da-a335-9cb35661ed1c"). InnerVolumeSpecName "kube-api-access-qj996". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:34:38.342309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.342284 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-xks-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:38.342309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.342306 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj996\" (UniqueName: \"kubernetes.io/projected/0276489b-29b0-46da-a335-9cb35661ed1c-kube-api-access-qj996\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:38.342440 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.342316 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0276489b-29b0-46da-a335-9cb35661ed1c-maas-api-openshift-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:38.996917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.996885 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" event={"ID":"0276489b-29b0-46da-a335-9cb35661ed1c","Type":"ContainerDied","Data":"1492abdeace1cdf2e2e12cbdf23eaa5e27195eb4282ba240e3590869e2b74276"} Nov 03 21:34:38.997078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.996929 4227 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1492abdeace1cdf2e2e12cbdf23eaa5e27195eb4282ba240e3590869e2b74276" Nov 03 21:34:38.997078 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:38.996931 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64" Nov 03 21:34:47.359313 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.359278 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8"] Nov 03 21:34:47.359891 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.359768 4227 cpu_manager.go:414] "containerMap: removing container" podUID="0276489b-29b0-46da-a335-9cb35661ed1c" containerName="revoke-keys" Nov 03 21:34:47.359891 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.359788 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0276489b-29b0-46da-a335-9cb35661ed1c" containerName="revoke-keys" Nov 03 21:34:47.360018 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.359902 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="0276489b-29b0-46da-a335-9cb35661ed1c" containerName="revoke-keys" Nov 03 21:34:47.363119 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.363096 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.461824 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.461781 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8"] Nov 03 21:34:47.462019 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.461953 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:34:47.462086 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.462063 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:34:47.462684 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.462644 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nqbhl\"" Nov 03 21:34:47.462850 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.462718 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:34:47.462850 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.462766 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 21:34:47.516199 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.516141 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-metrics-certs\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.516379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.516286 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7763260c-d524-4e80-adce-71a47ea996bf-openshift-service-ca\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.516379 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.516329 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn46l\" (UniqueName: \"kubernetes.io/projected/7763260c-d524-4e80-adce-71a47ea996bf-kube-api-access-gn46l\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.516479 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.516387 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-maas-api-tls\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.616971 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.616885 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7763260c-d524-4e80-adce-71a47ea996bf-openshift-service-ca\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.616971 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.616943 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn46l\" (UniqueName: \"kubernetes.io/projected/7763260c-d524-4e80-adce-71a47ea996bf-kube-api-access-gn46l\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.616971 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.616964 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-maas-api-tls\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.617251 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.617015 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-metrics-certs\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.617587 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.617564 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7763260c-d524-4e80-adce-71a47ea996bf-openshift-service-ca\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.619486 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.619463 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-maas-api-tls\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.619583 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.619504 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-metrics-certs\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.669657 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.669626 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn46l\" (UniqueName: \"kubernetes.io/projected/7763260c-d524-4e80-adce-71a47ea996bf-kube-api-access-gn46l\") pod \"maas-api-5ff6ff6f8b-fplh8\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:47.676082 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:47.676061 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:48.041588 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:48.041555 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8"] Nov 03 21:34:48.049278 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:34:48.049250 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7763260c_d524_4e80_adce_71a47ea996bf.slice/crio-8faa8573bba85d979f176d76d10db90b718a392c4d5c968fe63d0a49b748524a WatchSource:0}: Error finding container 8faa8573bba85d979f176d76d10db90b718a392c4d5c968fe63d0a49b748524a: Status 404 returned error can't find the container with id 8faa8573bba85d979f176d76d10db90b718a392c4d5c968fe63d0a49b748524a Nov 03 21:34:49.039371 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:49.039331 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" event={"ID":"7763260c-d524-4e80-adce-71a47ea996bf","Type":"ContainerStarted","Data":"bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768"} Nov 03 21:34:49.039905 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:49.039409 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:49.039905 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:49.039437 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" event={"ID":"7763260c-d524-4e80-adce-71a47ea996bf","Type":"ContainerStarted","Data":"8faa8573bba85d979f176d76d10db90b718a392c4d5c968fe63d0a49b748524a"} Nov 03 21:34:49.146722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:49.146595 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" podStartSLOduration=1.320681203 podStartE2EDuration="2.14657651s" totalImagesPullingTime="825.895307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:49.146064594 +0000 UTC m=+3956.383042470" watchObservedRunningTime="2026-11-03 21:34:49.14657651 +0000 UTC m=+3956.383554385" Nov 03 21:34:55.048577 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.048546 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:34:55.548717 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.548685 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz"] Nov 03 21:34:55.552219 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.552198 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.588006 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.587978 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tn5\" (UniqueName: \"kubernetes.io/projected/d9f41459-a7f8-4f36-9f82-5228e9a44510-kube-api-access-h9tn5\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.588127 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.588084 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f41459-a7f8-4f36-9f82-5228e9a44510-openshift-service-ca\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.588197 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.588121 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9f41459-a7f8-4f36-9f82-5228e9a44510-metrics-certs\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.588277 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.588253 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9f41459-a7f8-4f36-9f82-5228e9a44510-maas-api-tls\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.609031 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.608997 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz"] Nov 03 21:34:55.689258 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.689227 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f41459-a7f8-4f36-9f82-5228e9a44510-openshift-service-ca\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.689410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.689268 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9f41459-a7f8-4f36-9f82-5228e9a44510-metrics-certs\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.689410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.689304 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9f41459-a7f8-4f36-9f82-5228e9a44510-maas-api-tls\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.689410 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.689335 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tn5\" (UniqueName: \"kubernetes.io/projected/d9f41459-a7f8-4f36-9f82-5228e9a44510-kube-api-access-h9tn5\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.689889 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.689861 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f41459-a7f8-4f36-9f82-5228e9a44510-openshift-service-ca\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.691799 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.691775 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9f41459-a7f8-4f36-9f82-5228e9a44510-maas-api-tls\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.691904 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.691809 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9f41459-a7f8-4f36-9f82-5228e9a44510-metrics-certs\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.743732 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.743701 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tn5\" (UniqueName: \"kubernetes.io/projected/d9f41459-a7f8-4f36-9f82-5228e9a44510-kube-api-access-h9tn5\") pod \"maas-api-59fffb9f98-cvzmz\" (UID: \"d9f41459-a7f8-4f36-9f82-5228e9a44510\") " pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:55.864497 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:55.864399 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:56.080601 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:34:56.080563 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:34:56.230833 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:56.230795 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz"] Nov 03 21:34:56.237570 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:34:56.237539 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f41459_a7f8_4f36_9f82_5228e9a44510.slice/crio-aa822119b377dcf267fd8c7018000ac6f738e53d38cd567bafa215b0bc76ebae WatchSource:0}: Error finding container aa822119b377dcf267fd8c7018000ac6f738e53d38cd567bafa215b0bc76ebae: Status 404 returned error can't find the container with id aa822119b377dcf267fd8c7018000ac6f738e53d38cd567bafa215b0bc76ebae Nov 03 21:34:57.074937 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:57.074900 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" event={"ID":"d9f41459-a7f8-4f36-9f82-5228e9a44510","Type":"ContainerStarted","Data":"aa822119b377dcf267fd8c7018000ac6f738e53d38cd567bafa215b0bc76ebae"} Nov 03 21:34:58.080508 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:58.080468 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" event={"ID":"d9f41459-a7f8-4f36-9f82-5228e9a44510","Type":"ContainerStarted","Data":"1e18fbbf692b93c218449ae5b4b0383427881c8632c54e5b8dbfc5013570860d"} Nov 03 21:34:58.080941 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:58.080635 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:34:58.191118 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:34:58.190923 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" podStartSLOduration=2.330433071 podStartE2EDuration="3.190903955s" totalImagesPullingTime="860.470884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:58.190436804 +0000 UTC m=+3965.427414678" watchObservedRunningTime="2026-11-03 21:34:58.190903955 +0000 UTC m=+3965.427881826" Nov 03 21:35:04.090868 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.090834 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59fffb9f98-cvzmz" Nov 03 21:35:04.246112 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.246075 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8"] Nov 03 21:35:04.246374 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.246330 4227 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" podUID="7763260c-d524-4e80-adce-71a47ea996bf" containerName="maas-api" containerID="cri-o://bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768" gracePeriod=30 Nov 03 21:35:04.496606 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.496578 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:35:04.578303 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.578274 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7763260c-d524-4e80-adce-71a47ea996bf-kube-api-access-gn46l\" (UniqueName: \"kubernetes.io/projected/7763260c-d524-4e80-adce-71a47ea996bf-kube-api-access-gn46l\") pod \"7763260c-d524-4e80-adce-71a47ea996bf\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " Nov 03 21:35:04.578503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.578331 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-metrics-certs\") pod \"7763260c-d524-4e80-adce-71a47ea996bf\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " Nov 03 21:35:04.578503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.578373 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7763260c-d524-4e80-adce-71a47ea996bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7763260c-d524-4e80-adce-71a47ea996bf-openshift-service-ca\") pod \"7763260c-d524-4e80-adce-71a47ea996bf\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " Nov 03 21:35:04.578503 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.578414 4227 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-maas-api-tls\") pod \"7763260c-d524-4e80-adce-71a47ea996bf\" (UID: \"7763260c-d524-4e80-adce-71a47ea996bf\") " Nov 03 21:35:04.578795 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.578766 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7763260c-d524-4e80-adce-71a47ea996bf-openshift-service-ca" pod "7763260c-d524-4e80-adce-71a47ea996bf" (UID: "7763260c-d524-4e80-adce-71a47ea996bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:04.580587 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.580555 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-metrics-certs" pod "7763260c-d524-4e80-adce-71a47ea996bf" (UID: "7763260c-d524-4e80-adce-71a47ea996bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:35:04.580587 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.580573 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-maas-api-tls" pod "7763260c-d524-4e80-adce-71a47ea996bf" (UID: "7763260c-d524-4e80-adce-71a47ea996bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:35:04.580740 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.580628 4227 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7763260c-d524-4e80-adce-71a47ea996bf-kube-api-access-gn46l" pod "7763260c-d524-4e80-adce-71a47ea996bf" (UID: "7763260c-d524-4e80-adce-71a47ea996bf"). InnerVolumeSpecName "kube-api-access-gn46l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:04.679441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.679409 4227 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn46l\" (UniqueName: \"kubernetes.io/projected/7763260c-d524-4e80-adce-71a47ea996bf-kube-api-access-gn46l\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:04.679441 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.679440 4227 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-metrics-certs\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:04.679660 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.679450 4227 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7763260c-d524-4e80-adce-71a47ea996bf-openshift-service-ca\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:04.679660 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:04.679459 4227 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7763260c-d524-4e80-adce-71a47ea996bf-maas-api-tls\") on node \"ip-10-0-134-71.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:05.118531 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.118431 4227 generic.go:356] "Generic (PLEG): container finished" podID="7763260c-d524-4e80-adce-71a47ea996bf" containerID="bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768" exitCode=0 Nov 03 21:35:05.118531 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.118493 4227 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" Nov 03 21:35:05.119099 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.118520 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" event={"ID":"7763260c-d524-4e80-adce-71a47ea996bf","Type":"ContainerDied","Data":"bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768"} Nov 03 21:35:05.119099 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.118585 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8" event={"ID":"7763260c-d524-4e80-adce-71a47ea996bf","Type":"ContainerDied","Data":"8faa8573bba85d979f176d76d10db90b718a392c4d5c968fe63d0a49b748524a"} Nov 03 21:35:05.119099 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.118618 4227 scope.go:122] "RemoveContainer" containerID="bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768" Nov 03 21:35:05.127787 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.127768 4227 scope.go:122] "RemoveContainer" containerID="bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768" Nov 03 21:35:05.128078 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:35:05.128057 4227 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768\": container with ID starting with bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768 not found: ID does not exist" containerID="bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768" Nov 03 21:35:05.128129 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.128089 4227 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768"} err="failed to get container status \"bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768\": rpc error: code = NotFound desc = could not find container \"bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768\": container with ID starting with bbc2a3e05caf1a596037458f4e45424d74e8fbb333fb106b10bada6b71fa5768 not found: ID does not exist" Nov 03 21:35:05.270959 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.270920 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8"] Nov 03 21:35:05.274929 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:05.274903 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff6ff6f8b-fplh8"] Nov 03 21:35:06.897890 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:06.897858 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7763260c-d524-4e80-adce-71a47ea996bf" path="/var/lib/kubelet/pods/7763260c-d524-4e80-adce-71a47ea996bf/volumes" Nov 03 21:35:26.081036 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:35:26.081001 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:35:54.193697 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:35:54.193662 4227 scope.go:122] "RemoveContainer" containerID="e6e035c0149e8b2c96796460fae5d641125820ab26bec2167510f45d171a7280" Nov 03 21:35:56.081065 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:35:56.081025 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:36:26.081664 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:36:26.081624 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:36:56.081230 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:36:56.081174 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:37:26.081292 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:37:26.081255 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:37:56.080621 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:37:56.080578 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:38:26.081688 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:38:26.081649 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:38:32.042516 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:38:32.042484 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz"] Nov 03 21:38:32.088624 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:38:32.088595 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p8lzz"] Nov 03 21:38:32.898570 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:38:32.898540 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3351269f-df13-477f-a717-01507f976a00" path="/var/lib/kubelet/pods/3351269f-df13-477f-a717-01507f976a00/volumes" Nov 03 21:38:43.561417 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:38:43.561320 4227 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rh44r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:38:43.561417 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:38:43.561400 4227 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rh44r" podUID="5f02aedc-87ae-4383-a34c-40f3cbe4eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:38:53.418843 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:38:53.418817 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:38:53.433521 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:38:53.433497 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx8t5_0568938d-c9e2-4e37-b768-cdeddaacf33e/ovn-acl-logging/0.log" Nov 03 21:38:56.082533 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:38:56.082494 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:39:08.045912 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:39:08.045874 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd"] Nov 03 21:39:08.093924 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:39:08.093883 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sznqd"] Nov 03 21:39:08.899176 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:39:08.899130 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f71a28-de40-42ef-9c7b-6d04886a4f4f" path="/var/lib/kubelet/pods/31f71a28-de40-42ef-9c7b-6d04886a4f4f/volumes" Nov 03 21:39:26.080217 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:39:26.080138 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:39:39.043718 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:39:39.043681 4227 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64"] Nov 03 21:39:39.090887 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:39:39.090858 4227 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc19917-jtj64"] Nov 03 21:39:40.897629 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:39:40.897593 4227 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0276489b-29b0-46da-a335-9cb35661ed1c" path="/var/lib/kubelet/pods/0276489b-29b0-46da-a335-9cb35661ed1c/volumes" Nov 03 21:39:54.240387 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:39:54.240353 4227 scope.go:122] "RemoveContainer" containerID="7ebdf051cbc8cd41c66bf1db395e50c1a57a28a04709ef724085d956a8c974a1" Nov 03 21:39:56.080385 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:39:56.080344 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:40:05.209027 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:05.208952 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xjx5n_41e62745-afd6-45e5-b3fe-4b9c82669d5a/server/0.log" Nov 03 21:40:06.706843 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:06.706816 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59fffb9f98-cvzmz_d9f41459-a7f8-4f36-9f82-5228e9a44510/maas-api/0.log" Nov 03 21:40:11.588515 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:11.588478 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-twmjn_32c44223-9bfe-4f28-b55e-290e1f9f7baf/discovery/0.log" Nov 03 21:40:11.805124 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:11.805091 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bb87d7467-f6vbg_dba92d2a-f1bf-4af8-ad77-4918b95e6ad6/kube-auth-proxy/0.log" Nov 03 21:40:12.957854 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:12.957820 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-ljp6z_1f780b9d-4c02-4e0f-9cec-9429a0ee4619/istio-proxy/0.log" Nov 03 21:40:13.171085 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:13.171046 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-rh44r_5f02aedc-87ae-4383-a34c-40f3cbe4eef9/istio-proxy/0.log" Nov 03 21:40:13.616685 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:13.616656 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-skskh_566679d2-822c-4dfc-b266-459ba037cdba/istio-proxy/0.log" Nov 03 21:40:26.080875 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:40:26.080838 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:40:38.391865 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:38.391833 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l2jnc_71213df2-9380-4e6b-9971-73d2404030ae/manager/0.log" Nov 03 21:40:38.915445 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:38.915419 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xjx5n_41e62745-afd6-45e5-b3fe-4b9c82669d5a/server/0.log" Nov 03 21:40:40.214917 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:40.214886 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59fffb9f98-cvzmz_d9f41459-a7f8-4f36-9f82-5228e9a44510/maas-api/0.log" Nov 03 21:40:44.390715 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:44.390683 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-twmjn_32c44223-9bfe-4f28-b55e-290e1f9f7baf/discovery/0.log" Nov 03 21:40:44.606303 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:44.606265 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bb87d7467-f6vbg_dba92d2a-f1bf-4af8-ad77-4918b95e6ad6/kube-auth-proxy/0.log" Nov 03 21:40:45.710175 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:45.710130 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-ljp6z_1f780b9d-4c02-4e0f-9cec-9429a0ee4619/istio-proxy/0.log" Nov 03 21:40:45.921649 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:45.921617 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-rh44r_5f02aedc-87ae-4383-a34c-40f3cbe4eef9/istio-proxy/0.log" Nov 03 21:40:46.373798 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:46.373748 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-skskh_566679d2-822c-4dfc-b266-459ba037cdba/istio-proxy/0.log" Nov 03 21:40:54.259041 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:54.259008 4227 scope.go:122] "RemoveContainer" containerID="80e277ca98bdd18732cfacd3294ec991e11a27196d5fbb4c0e53f6d00fa614a1" Nov 03 21:40:54.266991 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:40:54.266971 4227 scope.go:122] "RemoveContainer" containerID="f0faea4e6abf0eb6134ef0cd59ba2ac4875d1e46ed19162f3a3bc1b5ec22de74" Nov 03 21:40:56.081497 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:40:56.081461 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:41:26.081320 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:41:26.081279 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:41:56.080746 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:41:56.080666 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:42:23.354135 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.354098 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dd722/must-gather-9c6vw"] Nov 03 21:42:23.354645 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.354448 4227 cpu_manager.go:414] "containerMap: removing container" podUID="7763260c-d524-4e80-adce-71a47ea996bf" containerName="maas-api" Nov 03 21:42:23.354645 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.354459 4227 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7763260c-d524-4e80-adce-71a47ea996bf" containerName="maas-api" Nov 03 21:42:23.354645 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.354555 4227 memory_manager.go:365] "RemoveStaleState removing state" podUID="7763260c-d524-4e80-adce-71a47ea996bf" containerName="maas-api" Nov 03 21:42:23.357629 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.357606 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:23.453115 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.453076 4227 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dd722\"/\"default-dockercfg-85g9m\"" Nov 03 21:42:23.453314 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.453179 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dd722\"/\"kube-root-ca.crt\"" Nov 03 21:42:23.453314 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.453188 4227 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dd722\"/\"openshift-service-ca.crt\"" Nov 03 21:42:23.454696 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.454679 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/must-gather-9c6vw"] Nov 03 21:42:23.457578 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.457557 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zk6\" (UniqueName: \"kubernetes.io/projected/37194d74-fe69-45b3-a16e-6ec1deff3f99-kube-api-access-w2zk6\") pod \"must-gather-9c6vw\" (UID: \"37194d74-fe69-45b3-a16e-6ec1deff3f99\") " pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:23.457640 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.457615 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37194d74-fe69-45b3-a16e-6ec1deff3f99-must-gather-output\") pod \"must-gather-9c6vw\" (UID: \"37194d74-fe69-45b3-a16e-6ec1deff3f99\") " pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:23.558199 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.558137 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zk6\" (UniqueName: \"kubernetes.io/projected/37194d74-fe69-45b3-a16e-6ec1deff3f99-kube-api-access-w2zk6\") pod \"must-gather-9c6vw\" (UID: \"37194d74-fe69-45b3-a16e-6ec1deff3f99\") " pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:23.558404 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.558271 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37194d74-fe69-45b3-a16e-6ec1deff3f99-must-gather-output\") pod \"must-gather-9c6vw\" (UID: \"37194d74-fe69-45b3-a16e-6ec1deff3f99\") " pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:23.558665 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.558639 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37194d74-fe69-45b3-a16e-6ec1deff3f99-must-gather-output\") pod \"must-gather-9c6vw\" (UID: \"37194d74-fe69-45b3-a16e-6ec1deff3f99\") " pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:23.611716 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.611647 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zk6\" (UniqueName: \"kubernetes.io/projected/37194d74-fe69-45b3-a16e-6ec1deff3f99-kube-api-access-w2zk6\") pod \"must-gather-9c6vw\" (UID: \"37194d74-fe69-45b3-a16e-6ec1deff3f99\") " pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:23.678807 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:23.678767 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/must-gather-9c6vw" Nov 03 21:42:24.045920 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:24.045888 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/must-gather-9c6vw"] Nov 03 21:42:24.053590 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:42:24.053558 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37194d74_fe69_45b3_a16e_6ec1deff3f99.slice/crio-9070133f168c931a109b864b0be214e347ca831ebbf200df5e3e4b8be06c2c33 WatchSource:0}: Error finding container 9070133f168c931a109b864b0be214e347ca831ebbf200df5e3e4b8be06c2c33: Status 404 returned error can't find the container with id 9070133f168c931a109b864b0be214e347ca831ebbf200df5e3e4b8be06c2c33 Nov 03 21:42:24.055460 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:24.055439 4227 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:42:25.047179 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:25.047116 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/must-gather-9c6vw" event={"ID":"37194d74-fe69-45b3-a16e-6ec1deff3f99","Type":"ContainerStarted","Data":"9070133f168c931a109b864b0be214e347ca831ebbf200df5e3e4b8be06c2c33"} Nov 03 21:42:26.055190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:26.054496 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/must-gather-9c6vw" event={"ID":"37194d74-fe69-45b3-a16e-6ec1deff3f99","Type":"ContainerStarted","Data":"1b15ef9e2e2856972ad0ad70a8f779100b8a8cc50cd8d215d7b0391e8481f6ca"} Nov 03 21:42:26.055190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:26.054553 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/must-gather-9c6vw" event={"ID":"37194d74-fe69-45b3-a16e-6ec1deff3f99","Type":"ContainerStarted","Data":"e769e41d010292bb69534a372fbe125ac669a1286035075e089c5309f70f6368"} Nov 03 21:42:26.087137 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:42:26.087088 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:42:26.161065 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:26.161006 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dd722/must-gather-9c6vw" podStartSLOduration=1.996601552 podStartE2EDuration="3.160991092s" totalImagesPullingTime="1.16438954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:26.159821787 +0000 UTC m=+4413.396799661" watchObservedRunningTime="2026-11-03 21:42:26.160991092 +0000 UTC m=+4413.397968962" Nov 03 21:42:33.588549 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:33.588516 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xfqt9_8eddcaf9-30b4-44e8-bc77-24b08f234497/global-pull-secret-syncer/0.log" Nov 03 21:42:36.402884 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:36.397081 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hc7pp_9a652af6-69fa-467b-abfe-bff66e3afbde/konnectivity-agent/0.log" Nov 03 21:42:37.174484 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:37.174434 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-71.us-west-2.compute.internal_b295d4bbd28e0091df21418ae66e44d9/haproxy/0.log" Nov 03 21:42:41.943239 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:41.943205 4227 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv"] Nov 03 21:42:41.949915 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:41.949891 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.032226 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.032192 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv"] Nov 03 21:42:42.041722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.041690 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-podres\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.041722 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.041732 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-proc\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.041956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.041813 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndsj\" (UniqueName: \"kubernetes.io/projected/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-kube-api-access-zndsj\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.041956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.041850 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-sys\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.041956 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.041913 4227 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-lib-modules\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143396 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143357 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-podres\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143412 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-proc\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143446 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndsj\" (UniqueName: \"kubernetes.io/projected/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-kube-api-access-zndsj\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143479 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-sys\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143487 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-podres\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143507 4227 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-lib-modules\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143560 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-proc\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143577 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-lib-modules\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.143586 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.143576 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-sys\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.197395 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.197318 4227 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndsj\" (UniqueName: \"kubernetes.io/projected/e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82-kube-api-access-zndsj\") pod \"perf-node-gather-daemonset-rfgdv\" (UID: \"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.269309 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.269273 4227 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:42.638231 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:42.638094 4227 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv"] Nov 03 21:42:42.645714 ip-10-0-134-71 kubenswrapper[4227]: W1103 21:42:42.645672 4227 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode3445b2c_34e2_49dd_94b8_9a5ff9ae0c82.slice/crio-c21279c6e07baa42f2fddf7f000808bcd7803490a21e123a27fd59159e928470 WatchSource:0}: Error finding container c21279c6e07baa42f2fddf7f000808bcd7803490a21e123a27fd59159e928470: Status 404 returned error can't find the container with id c21279c6e07baa42f2fddf7f000808bcd7803490a21e123a27fd59159e928470 Nov 03 21:42:43.175717 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:43.175679 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" event={"ID":"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82","Type":"ContainerStarted","Data":"1aff90f71a28ce19454eb2fc60483e9d99ebd8e07c69db64ba24f496346eb932"} Nov 03 21:42:43.176154 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:43.175750 4227 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" event={"ID":"e3445b2c-34e2-49dd-94b8-9a5ff9ae0c82","Type":"ContainerStarted","Data":"c21279c6e07baa42f2fddf7f000808bcd7803490a21e123a27fd59159e928470"} Nov 03 21:42:43.176154 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:43.175780 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:43.283533 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:43.283483 4227 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" podStartSLOduration=2.283468769 podStartE2EDuration="2.283468769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:43.282499076 +0000 UTC m=+4430.519476950" watchObservedRunningTime="2026-11-03 21:42:43.283468769 +0000 UTC m=+4430.520446690" Nov 03 21:42:46.525022 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:46.524995 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_005f73b8-626f-443f-a6c5-033d0d1be875/alertmanager/0.log" Nov 03 21:42:46.927243 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:46.927211 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_005f73b8-626f-443f-a6c5-033d0d1be875/config-reloader/0.log" Nov 03 21:42:47.333562 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:47.333475 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_005f73b8-626f-443f-a6c5-033d0d1be875/kube-rbac-proxy-web/0.log" Nov 03 21:42:47.737114 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:47.737087 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_005f73b8-626f-443f-a6c5-033d0d1be875/kube-rbac-proxy/0.log" Nov 03 21:42:48.140585 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:48.140505 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_005f73b8-626f-443f-a6c5-033d0d1be875/kube-rbac-proxy-metric/0.log" Nov 03 21:42:48.543190 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:48.543135 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_005f73b8-626f-443f-a6c5-033d0d1be875/prom-label-proxy/0.log" Nov 03 21:42:48.948380 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:48.948350 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_005f73b8-626f-443f-a6c5-033d0d1be875/init-config-reloader/0.log" Nov 03 21:42:49.190298 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:49.190273 4227 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-rfgdv" Nov 03 21:42:53.101563 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:53.101530 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jq2dt_4e664735-47ec-43df-8b7e-32b12315bf4e/node-exporter/0.log" Nov 03 21:42:53.509755 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:53.509726 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jq2dt_4e664735-47ec-43df-8b7e-32b12315bf4e/kube-rbac-proxy/0.log" Nov 03 21:42:53.921483 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:42:53.921453 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jq2dt_4e664735-47ec-43df-8b7e-32b12315bf4e/init-textfile/0.log" Nov 03 21:42:56.088026 ip-10-0-134-71 kubenswrapper[4227]: E1103 21:42:56.087986 4227 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wcndc" containerName="extractor" Nov 03 21:43:00.533390 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:43:00.533355 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-spxv8_71ff80c2-3f12-4450-9b85-32338106f953/prometheus-operator-admission-webhook/0.log" Nov 03 21:43:00.941499 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:43:00.941467 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bcf87d6c9-ck62k_08f9c3db-d42c-4367-891c-eca082d6e806/telemeter-client/0.log" Nov 03 21:43:01.344719 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:43:01.344636 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bcf87d6c9-ck62k_08f9c3db-d42c-4367-891c-eca082d6e806/reload/0.log" Nov 03 21:43:01.748311 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:43:01.748282 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bcf87d6c9-ck62k_08f9c3db-d42c-4367-891c-eca082d6e806/kube-rbac-proxy/0.log" Nov 03 21:43:04.479631 ip-10-0-134-71 kubenswrapper[4227]: I1103 21:43:04.479554 4227 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-twmjn_32c44223-9bfe-4f28-b55e-290e1f9f7baf/discovery/0.log"